A Closer Look at Transformers for Time Series Forecasting: Understanding Why They Work and Where They Struggle
Authors: Yu Chen
Nathalia Céspedes
Payam Barnaghi
Venue: ICML 2025 poster (2025)
Keywords: Time series forecasting
transformers
point-wise
patch-wise
variate-wise
tokenization
model evaluation
Primary area: deep_learning->sequential_models_time_series
TL;DR: This study investigates the effectiveness of basic transformer architecture for time-series forecasting to reveal what are the keys to the success of simpler architectures over more complex designs.

Abstract: Time-series forecasting is crucial across various domains, including finance, healthcare, and energy. Transformer models, originally developed for natural language processing, have demonstrated significant potential in addressing challenges associated with time-series data. These models utilize different tokenization strategies, point-wise, patch-wise, and variate-wise, to represent time-series data, each resulting in different scope of attention maps. Despite the emergence of sophisticated architectures, simpler transformers  consistently outperform their more complex counterparts in widely used benchmarks. This study examines why point-wise transformers are generally less effective, why intra- and inter-variate attention mechanisms yield similar outcomes, and which architectural components drive the success of simpler models. By analyzing mutual information and evaluating models on synthetic datasets, we demonstrate that intra-variate dependencies are the primary contributors to prediction performance on benchmarks, while inter-variate dependencies have a minor impact. Additionally, techniques such as Z-score normalization and skip connections are also crucial. However, these results are largely influenced by the self-dependent and stationary nature of benchmark datasets. By validating our findings on real-world healthcare data, we provide insights for designing more effective transformers for practical applications.

A Closer Look at Transformers for Time Series Forecasting: Understanding Why They Work and Where They Struggle
Yu Chen 12 Nathalia Céspedes 12
Payam Barnaghi 12
Abstract
1. Introduction
Time-series forecasting plays crucial role in numerous critical applications, from finance and healthcare ergy and transportation. Accurate predictions of future val- ues based on historical data are indispensable for informed decision-making and strategic planning across these do- mains. Building on their success in natural language pro- cessing, transformer models (Vaswani et al. 2017) have rapidly gained attention from researchers forecasting. Compared to RNN-based models (Hochreiter & Schmidhuber 1996 Rangapuram et al., 2018 Salinas et al. 2020), transformers are capable of capturing long- range dependencies and efficiently processing large-scale datasets, making them a promising solution for addressing common challenges time series data.
Time-series forecasting is crucial across various domains, including finance, healthcare, and en- ergy. Transformer models, originally developed for natural language processing, have demon- strated significant potential addressing chal- lenges associated with time-series data. These models utilize different tokenization strategies, point-wise, patch-wise, and variate-wise, to rep- resent time-series data, each resulting in different scope of attention maps. Despite the emergence of sophisticated architectures, simpler transform- ers consistently outperform their more com- plex counterparts in widely used benchmarks. This study examines why point-wise transform- ers are generally less effective, why intra- and inter-variate attention mechanisms yield similar outcomes, and which architectural components drive the success of simpler models. By ana- lyzing mutual information and evaluating models on synthetic datasets, we demonstrate that intra- variate dependencies are the primary contribu- tors prediction performance benchmarks, while inter-variate dependencies have minor impact. Additionally, techniques such as Z-score normalization and skip connections are also cru- cial. However, these results are largely influ- enced by the self-dependent and stationary nature of benchmark datasets. By validating our find- ings on real-world healtheare data, we provide insights for designing more effective transform- ers for practical applications.
time-series
Based on how tokens are represented in the attention mechanism, transformer- based models for time series can be categorized into three types: point-wise, patch-wise, and variate-wise wise (Wang et al., 2024b). In point-wise ap- proaches, the embedding of each token is derived from the values of all variates at a specific time step. Patch-wise methods, on the other hand, represent small segments of the time series as tokens, with each token's embedding gen- erated from segment of single variate window. variate-wise approaches take more global per- spective by treating the entire time series of a variate as a single token. A visualization of these three token types for time series data is shown in Figure] 1
fixed time
Recent research on transformers for time series forecast- ing has explored various tokenization strategies, with many models incorporating hybrid token scopes enhance per- formance. However, point-wise transformers are generally less competitive than patch-wise and variate-wise ones. In- terestingly, some transformers with simpler token scopes and attention mechanisms, such as iTransformer (Liu et al.] 2023) and PatchTST (Nie et al. (023). perform excep- tionally well on standard time series forecasting bench- marks (Wang et al., 2024b). Notably, iTransformer is variate- wise transformer with inter-variate attention, where attention is computed across different variates. In contrast, PatchTST is a patch-wise transformer with intra-variate at- tention, focusing on interactions between patches from the
'Imperial College London, Department Brain Sci- ences, London, W12 ONN, United Kingdom 'The UK Dementia Research Institute, Care Research and Tech- nology Centre, United Kingdom. Correspondence Yu Chen <yu.chen@* imporial.ac.uk>. Payam Barnaghi <p.barmaghi aeanmagh@@imperiaa.a....
Proceedings of the 1nnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

A Closer Look at Transformers for Time Series Forecasting: Understanding Why They Work and Where They Struggle
Point wise token
Patchhaisa token
Variate-wise token
variate
variate
variate
valuo
valuo
Rim
Figure 1: Demonstration of different token representations in time-series transformers. A point-wise wise token is formed by all variate values at one time step; patch- patch-wise token is formed by segment patch of single variate in one time window; a variate-wise token is the entire time series of a single variate.
same variate. Despite these fundamental differences, both models often achieve similar performance benchmark.
wire tokens due to the self-attention mechanism.
The skip-connection in encoder layers plays an crucial role for learning intra-variate patterns. A variale-independent decoder also helps the model focus on the intra-variate dependencies. These findings address the question 3 and 4.
This study seeks to address the following key questions:
1. Why point-wise transformers generally less competi- tive in time series forecasting?
3. A key factor that contributes to the success of models like i'Transfomeer and PatchTST is not the model's archi- tecture itself, but rather the use of Z-score normalization and denormalization for the model's input and output. This technique is partieularly effective when the variates are sta- tionary throughout the combined duration of the observa- tion and prediction periods. However, it may degrade model's forecasting performance when this assumption is violated.
2. Why do transformers with intra-variate attention and those with inter-variate attention perform similarly?
3. Why do transformers with basic attention mechanisms excel in time series forecasting?
4. Which components in the basic transformer's archi- tecture contribute most to the success in time series forecasting?
These findings also help explain certain phenomena ob- served the literature. (Tam et al. (024] found that pre- trained large language models are particularly useful for time series forecasting, broader context is not essential for capturing intra-variate patterns. However, this conclusion is based on commonly used benchmarks and may not hold for fundamentally dif- ferent datasets, such as event time series. Additionally, (Zeng et al. 2023) suggested that linear models can outper- form certain transformers on time series forecasting bench- marks, point point-wise transformers struggle to effectively capture intra-variate patterns.
To achieve this goal, we conducted a comprehensive anal- ysis of selected representative transformers for time se- ries forecasting (Table D To ensure a unified compari- son across models with varying architectures and attention mechanisms, introduce new metrics that estimate the mutual information within and between variates based on model's 's input and output, capturing both varratv and inter-variate information flows in a mgdel-agnostic manner. Beyond standard benchmark datasets from the literature, we also designed a set of synthetic datasets with controlled intra- and inter-variate dependencies systematically as- sess transformer performance across different conditions.
understanding variate's
Moreover, the findings of this study offer complementary perspectives to several prior works. For instance, (Zhao & Shen (024) introduced plugin method to identify and ex- ploit locally stationary lead-lag relationships between vari- ates improve forecasting performance. Their work fo- cuses specific type of inter-varien dependency and demonstrated modeling such relationships can be benefi- cial. Another relevant work, Reversible Instance Normal- ization (RevIN) (Kim et al. 2021). proposed a method sim- ilar to Z-score normalization to address distributional shifts in time series data. While we also examined Z-score nor- malization in this study, key difference is that RevIN incorporates learnable parameters within its normalization process, whereas the Z-score normalization in our study is the standard, non-learnable version. Notably, we observed
Our findings provide valuable insights for understanding how transformers work in time series forecasting:
1. We demonstrate that transformers with superior perfor- mance excel at capturing vatra-variate patterns, while inter- variate patterns play much smaller role in model predic- tions, even in transformers equipped with inter-varias at- tention mechanisms. This can be attributed to the fact that, in the majority of time series forecasting benchmarks, each variate is largely self-dependent. These findings provide valuable insights into questions and 2.
2. The basic transformer encoder is capable of capturing the intra-variate dependencies when using patch-wise and
2

---

A Closer Look at Transformers for Time Series Forecasting: Understanding Why They Work and Where They Struggle
that Z-score normalization degraded model performance on synthetic datasets which are non-stationary with mono- tonic trends, outcome that contrasts with observations in RevIN.
7). TimeXer (Wang et 2024a) TemeXer combines the ideas of PatchTST and i'Transformer to enhance its abil- ity of capturing both intra- and vater-variate dependencies. It learns vntra-variate attention via patch-wise tokens and inter-variate attention through variate-wise tokens. The output of its encoder layers is a concatenation of the out- puts from both attention mechanisms. It's decoder is the same as PatchTST and iTransformer.
2. Related work
We selected set of time series transformers in our analy- sis, which are representative of the different token scopes and attention mechanisms used for time series forecasting. These models are summarized in Table and more details are provided below:
Some other point-wise approaches, such Reformer (Ki-) taev et al. 2020), Pyraformer (Liu et al. 2022), and In- former (Zhou et al. 2021). were not included in our exper- iments. This is because they are generally perform worse and their architectures are less representative. Temporal Fusion Transformer (TFT) (Lim et al., [02T], however, is a hybrid model that incorporates variable selection and SSTM-based encoders prior to applying attention. As sult, its token representations are less explicit and not as directly comparable to the models we selected. Addition- ally, (Tan et al. (2024) introduces PAttn, a patch-wise model similar to PStchTST in both token scope and architecture. Therefore, we consider PatchTST representative model for PAum. Furthermore, this study focuses on lightweight transformers for time series forecasting and does not in- clude pretrained large language models.
1). Transformer Vasswani et al. 2017) The standard transformer architecture with point-wiis wise tokens.
2.) Autoformer (Wu et al., 2021) Autoformer introduces autocorrelation attention to capture
dependen- cies in time series. It decomposes the time series into trend and seasonal components using a moving average method, as in traditional time series models (Box et al. (015)) Ad- ditionally, a wose projection is added to each encoder layer to capture inter-variate dependencies. Since autocor- relation attention is computed over the entire time series, consider Autoformer as hybrid model that incorpo- rates variate-wise and point-wise tokens.
Traditional statistical models such as ARIMA (AutoRe- gressive Integrated Moving Average) (Box & Pierce 1970), were not considered in this study as we are focusing on transformer- basedd models. However, such methods can be useful for analyzing the properties of single variates and provide certain interpretability to time series forecasting.
3). FEDformer Zhou et al. 2022) FEDformer introduces frequency-enhanced attention to capture periodic patterns in time series data. Building on trend and seasonal de- composition, it adds frequency decomposition for seasonal components. The architecture of FEDformer closely re- sembles Autoformer, also incorporating point-wise pro- jection in its encoder layers.
Additionally, Qiu et al. (2024) introduced new benchmarks for time series forecasting, primarily focusing on intra- variate perspectives, which differ significantly from the scope of the synthetic datasets in our study.
4). Crossformer (Zhang& & Yan 2023) Crossformer intro- duces a Two-Stage Attention mechanism to capture both intra- and inter-variate dependencies. It employs hierarchi- cal patch-wist wise tokens for learning cross-time (intra-variate) and cross-dimension (inter-variate) attention efficiently.
3. Method
5). PStchTST (Nie et al., 2023) PatchTST uses a vanilla transformer encoder architecture but with patch-wise to- kens. It segments the time series along the time axis into patches and applies attention to these patches for each variate. The model is incapable of capturing inter-variate patterns, as its decoder, a linear projection layer, is also variate-independent.
Standard metrics for time series forecasting, such as mean squared error (MSE) and mean absolute error (MAF), are insufficient for evaluating a model's ability to learn intra- and variate patterns. To overcome this limitation, we propose new metrics that estimate the mutual information (ooerr 1999) between a model's input and output, captur- ing intra- and inter-variate information flows in model- agnostic way. These metrics facilitate comparisons across models with different architectures.
6). iTransformer (Liu et al. 2023) iTTrasfformrr uses the same encoder architecture as the vanilla transformer but with variate-wise tokens. It treats the entire time series of a variate as a single token, thereby learning inter- variate at- tention through the multi-head self-attention mechanism. It's decoder variate-indepennenn linear projection, the same as PSt
Notations: Let x = {xI, X2, XM3 be a multivariate time series data sample, where x; = {x1,1, 2..2..... 2t,r} is the i-th variate with T time steps. The goal of time series forecasting is to predict the future values of each variate in x based on the historical values. We denote the predicted values as x; ( T2, ... &., TRR In addition,
3

---

A Closer Look at Transformers for Time Series Forecasting: Understanding Why They Work and Where They Struggle
Table I: Selected transformer-based models for time series forecasting in our analysis. Some models are hybrid that combines multiple token scopes in the model architecture. We consider Autoformer and FEDformer as hybrid with point- wise tokens because they both include point-wise projection in their encoders. Z-norm indicates whether the model uses Z-score normalization and denormalization for the input and output.
Inter-variate attention
Variate-independent decoder
Point-wise
Patch-wise
Variate- -wise
Z-norm
Transformer
(Vawwani et al.
2017)
Autoformer
Wu et al.
2021
FEDformer
Zhou et al.
2022)
v
Crossformer
Zhang & Yan
2023
v
PatchTST Nie ct al.
2023)
v
iTransformer (Liu
2023
TimeXer (Wang et al.
2024a)
X/1- = (x,lVj # i) represents the data sample excluding the i-th variate.
retains the original value, and the remaining instances are generated by adding Gaussian noise of varying strengths to the original value.
To quantify the dependency of the model's prediction for the j-th variate (x)) on the i-th variate (x;), estimate the mutual information between them, conditioned on X/s:
When j i, we call 0+ the intra-variate mutual infor- mation score (Intra MI), and when + i, we call oi as the inter-variate mutual information score (Inter MI). For a time series with more than two variates, we define two met- rics for evaluating inter-variate mutual information across variates:
x/xxx /x)) (x,,,,,,,,,,, x/1)
x,,
(1)
Since the model's 's output is deterministic given a complete
input, the conditional entropy H(x;) ((; x/s) = 0. Hence:
(i) Average Inter MI (Avg Inter MI): The average of all inter-variate mutual information scores.
K
I (x,1) = H (x,)\//)
log Ui
(2)
Arm
M
Here k is the k-th time step of &j and we assume X1|x/s N (Pug of I is the identity matrix. o estimates the extent to which changes in the prediction of variate are
M(M 1)
(ii) Maximum Inter MI (Max Inter MI): The maximum
caused by changes in variate Mathematically, of is the
inter-variate mutual information score.
variance of the predictions of variate j conditioned on in- puts where all variates are held fixed except for variate i. It is straightforward to see that the mutual information ((x):///x/; can be evaluated by the standard deviation of &)lx/ Therefore, we define a mutual information score as below:
max Jij Vi,j E 21, ... M)itj 6
We propose Max Inter MI as a measure of the mutual in- formation captured by model between the most strongly interacting variates. This metric helps assess whether model is effectively learning inter-variate dependencies, particularly in cases where the number of variates is large and only a few exhibit strong interactions. For instance, in Figure + Crossformer achieves the highest Max Inter MI on the Traffic dataset (which has 862 variates), while its average Avg Inter MI remains lower than that of most other models.
K
(12,,,, M) (3)
1=1
Where is the number of samples, K is the predicted se- quence length. Unlike Pearson's 's correlation, it is more ver- satile it captures both linear and non- non-linear relationships.
To estimate this conditional variance, the variation of vari- ate is introduced through N different samples by augment- ing original samples in the dataset. Specifically, each origi- nal sample is augmented into N 5 versions, differing only in the value of variate i: one instance is set to zero, one
4. Experiments
We conducted a series experiments to analyze the behav- ior of selected transformers on both real-world and syn-
4

---

A Closer Look at Transformers for Time Series Forecasting: Understanding Why They Work and Where They Struggle
thetic datasets. The experiments were implemented us- ing the Time Series Library (Wang al., 2024b). To ensure consistency, we employed a unified configuration across all prediction lengths within a given dataset for each model. As a result, the reported performance may differ from that presented in the original papers for these mod- els. All the experimental results in this work are averaged over 3 runs with different random seeds. Standard devia- tion of the results are provided in the Appendix. Code are available here: https: /tttp:://githbb. www.oo/c/o/00 TimeSeries- Transssssrrafffoooreee Ana lysis
Variato, a
Variate
Variata b
Variate b
24
48
120
144 168 192
24 48
P6 120
168 182
Time steps
Time stops
(a) 7 = 0.95, ax
(b) 7 = 0.5, a =0 0
4.1. Datasets
Variate a
1MMM/MM MMM M Mo M
Variate a
00
31
a.2
3.0
a1
3.2
Variate
Variate
b0
b2
b1
b2
120 268 192
120 164 105 192 Time stops
Time steps
(a) Independent variates.
(b) Dependent variates.
(c) = 0.95, = 0.8
(d) = 0.5, a = 0.8
Figure 2:
Demonstration of synthetic datasets with inde- pendent and dependent variates.
Figure 3: Visualization of synthetic data with different con- figurations of y and a. Figures Ba and 3b show indepen- dent variates with high and low autocorrelation, while Fig- 3r and [3d depict dependent variates with high and low autocorrelation.
We included the following datasets in our experiments, as they are among the most commonly used benchmarks in the literature: Weather, Electricity, Traffic, and ETT (com- prising four subsets). More details of these datasets are provided in Table the Appendix.
(2) Next, We generated two seasonal components for each variate, defined by specified amplitudes and frequencies. Each seasonal component was represented as a sine wave. The final time series was generated by adding the trend and seasonal components.
Additionally, we designed set of synthetic datasets with controlled intra- and inter-variate dependencies to evalu- ate the performance of different transformers under various conditions. Each synthetic dataset consists of two variates and is generated as hourly time series over a one-year pe- riod. Based on the relationship between the variates, we classify these datasets into two categories: independent and dependent (Figure 2
For the dependent variates, we initially generated two in- dependent variates using the above process, An additional step was then applied to introduce a specified dependency parameter (a E [0, 1) between the variates. This parame- ter controls the strength of the dependency: when a O, the variates were generated independently, with no depen- dency between them.
For the independent variates, we generated them inddpen- dently with the following process:
(1) We begin by generating trend component charac- terized by a specified autocorrelation strength parameter (7 E [o, I). Within the initial time window, the trend fol- lows a linear pattern, either increasing or decreasing. A random noise sampled from normal distribution was added the trend component. The formulation is as be- low and the time lag was fixed at k- = 24 in all synthetic datasets:
b = =a Σ W, + (1 a)b T=
(5)
In our experiments, we selected E {0.5, 0.95} to rep- resent low and high levels of autocorrelation, respectively. and a E {0, 0.2, 0.4, 0.8} to capture varying degrees of (4) dependency between variates. Figure illustrates the syn- thetic signals generated with different levels of 7 and a.
a 7
Σ T=t-k
a,a, + (1 7)aa,
5

---

A Closer Look at Transformers for Time Series Forecasting: Understanding Why They Work and Where They Struggle
MAF
Intra MI
Avg. Inter Mi
Max Inter Mi
04
2.250
0.2
0.225
Timelee
Astoronmot
Crosttemen
Patonnty
Towater
Brandoome
Peceommer
Peeests
mandoree
Timalen
Autotomee
Tometo
MAE
Intra
Avg. Inter
Max Inter MI
0,010
0.4
tratfic
0.008
0.12
0.1
0.002
Pacoryy
Towetor
Thandoomer
macen
Timelee
Quantamer
MAE
Intra
Avg. Inter
Max Inter
weather
0.5
0:075
0.050
3.4
0.2
0.200
Wanntoonat
Tomeker
Dewnttmman
Wendomee
Crocelan Croselemar
Paeettrr
Prediction length
Figure 4: Performance comparison of selected transformer-based models commonly used benchmark datasets.
Synthetic (y 0.95,
Synthetic (y- 0.95, d 0.2)
Synthetic (y 0.95, a 0.4)
Synthatic (y 0.95, a 0.8)
0.46
0.40
Trantermee
Autafommar
ventorme
Chosstorter
mondfomeer
nandforer
Auntferme
Crosffemer
maautormer
Timater
mendammer
Autaformer
REvermee
crosatomer
Pescersy
mendforer
Troater
Ronstonther
Autalormor
Penformer
Chasttmmer
Patch
Prenitatto
Time Kan
Synthetic (y a
Synthetic (y a 0.21
Synthetic ly = 0.5, a 0.4)
Synthetic (y 0.5, 2 0.8)
618
0.7
Transtermer
Autaforar
rEaforme
Croastomter
RACATET
Timekon
Tannforer
Autsfemerr
rEDiomer
croafommer
PatckTST
manntommer
Timater
Autiformer
Reveomee
crotalomer
mendfomer
Timater
Transtorter
Astuformr
PED rentsrme
Chosttomer
RatcATET
Prnestamtr
Timeko
Prediction length
Figure 5: Forecasting performance of selected transformer-based models synthetic datasets with varying degrees of
dependency between and within variates.
4.2. Results
ers with intra-variate attention and those with inter- variate attention often achieve similar performance?
In this section, we present and analyze the results of our ex- periments, organized around the research questions posed in this study.
We begin by comparing the performance of selected transformer-based models on widely used benchmark datasets (Figure 4. The full results all benchmarks datasets are provided in Figure of the Appendix. These results indicate that transformers employing point-wise to-
Why wise transformers are generally less com- petitive in time series forecasting? And why transform-
6

---

A Closer Look at Transformers for Time Series Forecasting: Understanding Why They Work and Where They Struggle
Table 2: Correlation between MAE and mutual information scores across various datasets. Additionally, the table includes the average correlation between different variates within each dataset (denoted as Avg Var CORT)
Synthetic
0.95)
Synthetic (7
0.5)
ETTh1
ETTh2
ETTml
ETTm2
Weather
Traffic
Electricity
a= 0
=
0.2
0.4
= 0.8
=
0.2
0.4
= 0.8
Avg Var Corr
0.222
0.325
0.224
0.324
0.296
0.564
0.489
0.017
0.034
0.061
0.048
0.016
0.012
0.019
0.031
MAE Intra
0.875
0.935
.0.94
.0.781
-0.715
.0.77
.0.739
.0.742
-0.515
-0.6
0.089
0.8
0.792
0.725
0.695
MAE- Avg Inter
0.616
0.725
0.477
0.656
0.376
0.266
0.596
0.681
0.728
0.788
0.292
0.457
0.457
0.54
0.12
MAE - Max Inter
0.64
0.784
0.544
0.802
0.32
0.11
0.347
0.681
0.728
0.788
0.292
0.457
0.457
0.54
0.12
kens are generally less effective at capturing patterns within a single variate, as evidenced by their lower Intra MI scores. In contrast, these models are more sensitive to inter- variate influences, leading to higher Avg Inter MI and Max Inter MI scores. However, despite these higher Inter MI scores, they do not significantly improve the forecasting performance. This suggests that the limited performance of point-wise transformers is primarily due to their inabil- ity to effectively capture intra-variate patterns. Addition- ally, these results indicate that the benchmark datasets are largely self-dependent, with variates showing minor inter- variate dependencies.
Furthermore, the average correlation between variates within each dataset (Avg Var Corr) is not a reliable indi- cator of the dependency strength between variates. This is evident in the synthetic datasets, where the correlation be- tween variates remains close to zero and is unaffected by the dependency strength parameter This also explains why datasets with higher variate correlation do not neces- sarily exhibit stronger inter-varian dependencies.
Norm
To validate this hypothesis, we conducted experiments synthetic datasets with varying levels of intra- and inter- variate dependencies. The results in Table Aebeall a strong negative correlation between MAE and Intra MI scores on benchmark datasets, as well as on synthetic datasets with high autocorrelation (7 0.95) and low inter-variate de- pendencies (a < 0.4). In contrast, on synthetic datasets with low autocorrelation (- = 0.5), the results show a pos- itive correlation between MAE and Intra MI scores. MSE follows the same pattern MAE, so it is omitted here for brevity. Accordingly. transformers that perform exception- ally well on benchmark datasets are less effective on syn- thetic datasets characterized by high inter-variate depen- dencies (ax 0.8) low autocorrelation (7 = 0.5). As illustrated in Figure 5 Crossformer outperforms PatchTST, iTransformer, and TimeXer in these scenarios. These find- ings suggest that the performance of transformers in those benchmarks of time series forecasting is heavily influenced by the strength of intra-variate patterns. As a result, point-wise transformers are less competitive due to their limited ability capture such patterns.
Altention
Skin
Encoder buyer
Conmection
projection
projection
Decoder
Xmn
Figure 6: Demonstration of iTransformer architecture
2. Why do transformers with basic attention mecha- nisms excel in time series forecasting? Which compo- nents in the basic transformer's architecture contribute most to the success in time series forecasting?
Given the importance of intra-variate patterns in time se- ries forecasting, we hypothesize that the key components of transformer's 's architecture are those that most effectively enhance its ability to capture intra-variate dependencies. To validate this hypothesis, we conducted an ablation study on the i'Tranffrmer (Liu et al. 2023) model, which includes the vanilla transformer encoder and a variant-independent linear decoder (Figure 6 PatchTST (Nie et al.) 2023) has similar architecture iTransformer, but its atten- tion mechanism operates within each variate rather than across all variates, forcing it to focus entirely on capturing intra-variate patterns. In our experiments, we modified the model by removing the skip connections in the encoder lay- ers and/or replacing the puriate-independee decoder with a variate-dependent one. Specifically, for the benchmark
We also observe that, despite having inter-variate attention mechanisms, iTransformer and TimeXer generally exhibit lower Inter MI scores compared to other models (Figure This suggests that inter-variate patterns have a limited impact on forecasting benchmarks, offering insight into the second research question: Why do transformers with inter-variate attention often perform similarly to those with intra-variate attention? The reason is that both primarily focus on capturing intra-variate patterns.
7

---

A Closer Look at Transformers for Time Series Forecasting: Understanding Why They Work and Where They Struggle
Table 3: Ablation study on the iTransformer model architecture. "w/o SC" refers to the model without skip connections in the encoder layers, "VD-De" denotes the use of a variate- aariate-dependent decoder, "w/o SC & VD-De" indicates both mod- ifications applied, and "Original" represents the unmodified iTransformer model. The results are averaged over different prediction length, best results are highlighted in red.
Syathetic (y=0099)
Synthetic (y=00.5)
Metric
iTranffrrmer
Weather
ETThi
ETTh2
ETTml
ETTm2
Electricity
Traffic
c 0
a = 0.2
a = 0.4
( 0.8
< =
0 = 0.2
a 0.4
a = 0.8
w/o SC
0.295
0.472
0.420
0.421
0.340
0.320
0.591
0.443
0.436
0.434
0.455
0.789
0.789
0.798
0.764
VD-De
0.282
0.462
0.417
0.413
0.338
0.268
0.29
0.419
0.428
0.430
0.461
0.788
0.788
0.795
0.761
MAE
SC VD-De
0.287
0.423
0.424
0.342
0.363
0.631
0.419
0.449
0.789
0.799
0.763
Original
0.280
0.452
0,407
0.412
0.335
0.266
0.283
0,418
0.428
0.430
0.456
0.789
0.789
0.797
0.765
w/o SC
0:275
0.493
0.397
0.426
0.297
0.229
1.017
0.315
0.298
0.297
0.326
0.969
0.978
0.995
0.918
VD-De
0.257
0:473
0.397
0.411
0.293
0.173
0.425
0.276
0.286
0.291
0.336
0.969
0.973
0.987
0.912
MSE
w/o SC & VD-De
0.263
0.536
0.401
0.429
0.299
0.283
1.121
0,275
0.285
0.290
0.318
0.969
0.977
0.995
0.916
Original
0.26
0.46
0.383
0.409
0.291
0.175
0.422
0.274
0.286
0.290
0,330
0.970
0.978
0.993
0.921
datasets, we added a 2D convolutional layer before the variate-independent linear projection in the decoder. While for the synthetic datasets, we added a flattened linear layer. These changes were introduced to enable interactions be- tween variates in the decoder.
because the normalization is incompatible with their archi- tecture, which will be applied along the time dimension instead of the variate dimension. As shown in Table Z-score normalization has the opposite effect on synthetic datasets compared benchmark datasets. This suggests that the benchmark datasets are largely stationary, making it unreliable to evaluate models solely based on their per- formance on these datasets. Such comparisons may not ac- curately reflect their ability to handle non-stationary data. Therefore, the use of such normalization techniques should be assessed on a case-by-case basis, depending on the spe- cific characteristics of the dataset.
The results in Table 3 show that removing skip connec- tions notably degrades the model's performance across all benchmark datasets, with particularly significant degrada- tion on Electricity (MAE increases from 0,266 to 0.320) and Traffic (MAE increases from 0.283 to 0,591). How- ever, on the synthetic datasets, this degradation becomes negligible while the self-dependency becomes less (i.e. the autocorrelation y becomes lower, the inter-variate de- pendency a becomes higher). This observation suggests that skip connections are crucial for capturing intra- variate dependencies but may limit the model's abil- ity to capture inter-variate patterns. On the other hand, replacing the variate-independent decoder with a variate- dependent one has minor impact on performance for bench- mark datasets. However, it improves performance on syn- thetic datasets with high vater-variate dependencies (a 0.8) under low autocorrelatin (7 = 0.5). This indi- cates that the variate-dependent decoder enhances the model's capacity to capture inter-variate interactions, especially in scenarios with strong inter-variate depen- dencies. A more advanced decoder design could poten- tially further enhance performance on datasets with strong inter-variate dependencies, which we leave direction for future work.
5. Real-world Applications
Our experiments revealed that the performance of trans- formers in time series forecasting benchmarks rely heav- ily on their ability to capture intra-variate patterns. We also found that current benchmark datasets are largely self- dependent and stationary, with minor inter-variate depen- dencies. As result, models that excel on these datasets are less effective on synthetic datasets with strong inter-variate dependencies or low autocorrelation.
In real-world applications, time series data often exhibit more complicated patterns with varying degrees of inter- variate dependencies and autocorrelation. To validate our findings, we examined two worl-world healthcare datasets, MINDER and TIHM (Palermo al. (023). which contain location-activity data from PLwD (People Living with De- mentia). Understanding and forecasting behavioral pattern changes in dementia patients is crucial. However, these datasets differ significantly from both benchmarking and synthetic datasets, creating challenges for state-of-the-ar time series forecasting models.
Additionally, we observed that applying Z-score normal- ization and denormalization to the model's input and output significantly improves performance on benchmark datasets. To evaluate this, we tested four models, Cross- former, PatchTST, iTransformer, and TimeXer, with and across all
without Z-score normalizationddenrrmaiizzz datasets. Point-wise models were excluded from this test
Table (uble|5shooe shows the correlation between the forecasting met- ric MAE and mutual information scores on the MINDER
8

---

A Closer Look at Transformers for Time Series Forecasting: Understanding Why They Work and Where They Struggle
Table 4: The effect of Z-normalization on the performance of various models across benchmark and synthetic datasets. Results are averaged over different prediction lengths, with the best results highlighted in red. The blue bold font in the "Z.-Nomm" column indicates the original model implementation.
Synthetic (y = =0.55)
Synthetic (? =y= 0.5)
Metric
Model
Narm
Weather
ETTh1
ETTh2
ETTmll
ETTm2
Electricity
Traffic
=
a = 0.2
a = 0.4
Or 0.8
( =0=0
a= 0.2
= 0.4
= 0.8
w/
0.271
0.447
0.424
0.401
0.330
0.264
0,288
0.424
0.433
0.434
0.452
0.787
0.788
0.797
0.765
Cressormer
w/o
0.311
0.536
0.997
0.546
0.803
0.283
0.291
0.419
0.427
0.427
0.427
0.779
0.784
0.791
0.749
0.279
0.409
0.402
0.331
0.297
0.309
0.422
0.433
0.790
0.790
0.797
0.772
PatchTST
w/o
0.520
0.434
0.405
0.299
0.316
0.783
0.785
0.756
MAE
0.28
0:452
0.407
0.412
0.335
0.266
0.282
0.418
0.428
0.430
0.456
0.789
0.789
0.797
0.765
iTransformer
w/o
0.299
0.495
0.668
0.457
0.573
0.282
0.315
0.414
0.423
0.423
0.429
0.781
0.784
0,792
0.751
w/ 0.272
0.448
0.404
0.398
0.323
0.269
0.288
0.421
0.431
0.433
0.455
0.788
0.788
0.796
0.764
TimeXer
w/o 0.311
0.493
0.891
0.473
0.715
0.286
0.326
0.416
0.425
0.426
0.429
0.780
0.783
0.789
0.750
0.407
0.394
0.286
0.171
0.471
0.281
0.292
0.321
0.966
0.976
0.992
Crossformner w/o 0.256
0.572
1.742
0.571
1.715
0.189
0.566
0.275
0.285
0.288
0.288
0.946
0.96
0.977
0.883
w/ 0.256
0.453
0.386
0.388
0.286
0.208
0.482
0.280
0.291
0.294
0.322
0.973
0.981
0.992
0.940
PatchTST
w/o 0.246
0..489
0.600
0.423
0.379
0.204
0.595
0.273
0.281
0.284
0.289
0.953
0.967
0.978
0.899
MSE
w/ 0.260
0.460
0.383
0.409
0.291
0.175
0.422
0.274
0.286
0.290
0.330
0.970
0.978
0.993
0.921
iTransformer
w/o
0.250
0.508
0.842
0.450
0.653
0.182
0.571
0.268
0.279
0.282
0.292
0.950
0.964
0.978
0.888
0.242
0.455
0.379
0.384
0.276
0.170
0.470
0.279
0.291
0.295
0.327
0.968
0.975
0.989
0.921
TimeXer
1.555
0.467
1.156
0.187
0.604
0.291
Table 5: Correlation between MAE and mutual information scores on real-world health care datasets.
approaches.
Moreover, the results in Table show that Z-normalization has opposing effects MAE and MSE for the TIHM dataset, with MSE deteriorating while MAE improving. This effect likely arises because normalization alters the scale and distribution of the input data, which may disrupt the model's ability to capture larger errors while making it more effective in reducing absolute errors. These find- ings highlight the importance of carefully selecting evalu- ation metrics based application requirements, as differ- ent metrics may prioritize different aspects of forecasting performance, leading to varying interpretations of model effectiveness.
Avg Var Cor
MAE- Intra
MAE-A Avg Inter
MAB-Max Inter
MINDER
0.185
.0.358
0,882
0.873
THIM
0.321
.0.703
-0.11
-0.05
Table 6: Forecasting performance of I'Transformer variants on TIHM and MINDER datasets.
MAE
MSE
iTransformer
Original
VD-De
Original
VD-De
TIHM
0.623
0.610
0.835
0.807
MINDER
0.419
0.431
0.721
0.726
Table 7: Z-normalization shows opposing effects on MAE and MSE for TIHM dataset.
and TIHM datasets across all selected models. Despite the two datasets having similar variates, their properties dif- fer significantly, likely because the data was collected from different participants. Based on these correlations, we hy- pothesized that models with a stronger capacity for captur- ing inter-variate dependencies would perform better on the TIHM dataset. The results in Table support this hypoth- esis, showing that incorporating variate-dependent de- coder enhances the iTransformer model's performance the THIM dataset but negatively impacts its performance on the MINDER dataset. These findings suggest that a sin- gle, universal model architecture may not be suitable for all datasets or upplications, as different datasets exhibit dis- tinct dependency structures and require tailored modeling
Model
Crossformer
PatchTST
iTransformrr
TimeXer
Z-Norm
w/o
wlo
w/
w/o
MAB
0.739
0.748
0.612
0.614
0.622
0.634
0.61
0.614
TIHM
1.038
1.011
0.82
0.834
0.817
Future research should explore adaptive evaluation tech- niques and dynamic modeling of intra- and inter-variate dependencies. Expanding benchmarks with diverse, non- stationary datasets will further enhance model generaliz- ability in real-worke applications.
9

---

A Closer Look at Transformers for Time Series Forecasting: Understanding Why They Work and Where They Struggle
Acknowledgements
Liu, S., Yu, H., Liao, C., Li, Lin, W., Liu, A. X., and Dustdar, S. Pyraformer: Low-complexity pyramidal at- tention for long-range time series modeling and forecast- ing. In International Conference on Learning Represen- tations, 2022. URL forum? -eddddddddtttt 20n50EEXm
This research is supported by the UK Dementia Research Institute [award number UK DRI-7002] through UK DRI Ltd. principally funded by the Medical Research Council (MRC), and the UKRI Engineering and Physical Sciences Research Council (EPSRC) PROTECT Project (grant num- ber: EP/W031892/I). Infrastructure support for this re- search was provided by the NIHR Imperial Biomedical Research Centre (BRC) and the UKRI Medical Research Council (MRC). P.B. is also funded by the Great Ormond Street Hospital and the Royal Academy of Engineering (grant number: DBSRF2324-18-699 We would also like to thank our reviewers for their valuable comments and sug- gestions, which helped improve the quality and clarity of this manuscript.
rww.ee...w........ ev. net./
Liu, Y. Hu, T., Zhang, H., Wu, H., Wang, S., Ma, L., and Long, M. itransformer: Inverted transformers are effective for time series forecasting. arXiv preprint arXiv:3310.0660 2023.
Nie, Y., Nguyen, N. H., Sinthong, P., and Kalagnanam, J. A time series is worth 64 words: Long-term forecasting with transformers, In The Eleventh International Confer- ence on Learning Representations, 2023. htthttss: //openreview net htr.//ormmmm/m http://orunnniidddddoooooooo
Impact Statement
Palermo, F., Chen, Y., Capstick, A., Fletcher-Loyd, N.,
This paper presents work whose goal is advance the field of Machine Learning. There are several potential societal benefits of our work and potential risks which are mitigated in our clinical study, none which we feel must be specifi- cally highlighted here.
Walsh, C., Kouchaki, S., True, J., Balazikova, O., Soreq, E., Scott, G., et al. Tihm: An open dataset for remote healtheare monitoring in dementia. Scientific data, 10 (1):606, 2023.
Qiu, X., Hu, J., Zhou, L., Wu, X., Du, J., Zhang, B., Guo, C., Zhou, A., Jensen, C. S., Sheng, Z., et al. Tib: Towards comprehensive and fair benchmarking of time series forecasting methods. arXiv preprint arXiv:2403.201500 2024.
References
Box, G. F. and Pierce, D. A. Distribution of residual auto- correlations in autoregressive-interrace moving average time series models. Journal of the American statisticul Association, 65(332):1509-1526 1970.
Rangapuram, S. S., Seeger, M. W., Gasthaus, J., Stella, L., Wang, Y., and Januschowski, T. Deep state space models for time series forecasting. Advances in neural informa- tion processing systems, 31, 2018.
Box, G. E., Jenkins, G. M., Reinsel, G. C., and Ljung, G. M. Time series analysis: forecasting and control. John Wiley & Sons, 2015.
Salinas, D., Flunkert, V., Gasthaus, J., and Januschowski, T. Deepar: Probabilistic forecasting with autoregressive recurrent networks. International jourrnaal of forecasting, 36(3):1181-1191. 2020.
Cover, T. M. Elements of information theory. John Wiley & Sons, 1999.
Hochreiter, S. and Schmidhuber, J. Lstm can solve hard long time lag problems. Advances in neural information processing systems, 9, 1996.
Tan, M., Merrill, M. A., Gupta, V., Althoff, T., and Hartvigsen, T. Are language models actually useful for time series forecasting? In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Kim, T., Kim, J., Tae, Y., Park, C., Choi, J.-H., and Choo, Reversible instance normalization for accurate time- series forecasting against distribution shift. In Interna- tional conference learning representations, 2021.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. u., and Polosukhin, Attention is all you need. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL ht
Kitaev, N., Kaiser, L., and Levskaya, A. Reformer: The efficient transformer. In International Conference Learning Representations, 2020.
Lim, B., Ank, S. Ö., Loeff, N., and Plister, T. Temporal fusion transformers for interpretable multi-horizon time series forecasting. International Journal of Forecasting, (144)):7788177)) 2021.
cc cc/paper Bt.pppppp/pprcceiisss..eee
3 115e2299676eee1ffo055ldddddd3
pdf
10

---

A Closer Look at Transformers for Time Series Forecasting: Understanding Why They Work and Where They Struggic
Wang, S., Wu, H., Shi, X., Hu, T., Luo, H., Ma, L.., Zhang, J. Y., and Zhou, J. Timemixer: Decomposable multi- scale mixing for time series forecasting. arXiv preprint arXiv:2405. 14616, 2024a.
Noi.org/100.o
The MIN-
DER dataset is not publicly available due to privacy proto- cols and has a format and variates similar to those of the TIHM dataset.
Wang, Y., Wu, H., Dong. J., Liu, Y., Long, M., and Wang, J. Deep time series models: A comprehensive survey and benchmark. arXiv preprint wrXiv:2407.13278, 2024b.
A.I.I. REAL-WORLD HEALTHCARE DATASETS
TIMH dataset The TIHM dataset consists of five inter- connected tables, Activity, Sleep, Physiology, Labels, and Demographics, capturing various aspects of remote health- care monitoring. Each table includes timestamps and par- ticipant UUIDs for cross-referencing and synchronization. The dataset includes 56 participants, all over 50 years old with a verified diagnosis of dementia or mild cognitive im- pairment, who provided informed consent. Each partici- pant had a caregiver or study partner, and individuals with severe psychiatric conditions or terminal illnesses were ex- cluded. The study recorded average of 50 days of data per participant.
Wu, H., Xu, J., Wang, J., and Long, M. Autoformer: De- composition transformers with auto-correlation for long- term series forecasting. Advances in neural information processing systems, 34:22419-22438 2021.
Zeng, A., Chen, M., Zhang, L., and Xu, Q. Are transform- ers effective for time series forecasting? In Proceed- ings of the AAAI conference on artificial intelligence, volume 37, pp. 11 1111--11,28, 2023.
Zhang, Y. and Yan, Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting. In The eleventh international conference learning representations, 2023.
For this study, we selected the participant with the longest recording, spanning three months from April 1, 2019, to June 30, 2019. We focused on the Activity table, which tracks in-home movement using motion and door sensors. Specifically, the dataset used in this study includes activ- ity records from eight locations: back door, bathroom, bed- room, fridge door, front door, hallway, kitchen, and lounge. The data was aggregated on an hourly basis by summing the activity counts for each location.
Zhao, L. and Shen, Y. Rethinking channel dependence for multivariate time series forecasting: Learning from leading indicators. In The Twelfth International Confer- ence on Learning Representations, 2024. httss //open //openreview. net/ um dd=ji TVECUO+S
Zhou, H., Zhang. S., Peng. J., Zhang, S., Li, J., Xiong, H., and Zhang, W. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceed- ings of the AAAI conference on artificial intelligence, volume 35, pp. 11106-11115,003
MINDER dataset The MINDER dataset, an ongoing project, collects in-home data from 117 participants over the age of 50, all with a clinical diagnosis of dementia or mild cognitive impairment (MCI) Each participant had re- ceived either past or ongoing psychiatric treatment. For this study, we selected the participant with the longest recording, spanning two years from December 1, 2022, to November 30, 2024. The MINDER dataset is similar in format to TIHM, and we also focused exclusively on the Activity table. Specifically, the dataset for this work includes activity records from eight locations: WC, bath- room, bedroom, hallway, front door, kitchen, and lounge.
Zhou, T., Ma, Z. Wen, Q., Wang, X., Sun, L., and Jin, R. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. In Inter- national conference on machine learning, pp. 27268- 27286. PMLR, 2022.
A. Appendix
A.2. Full results of all selected models on benchmark datasets and synthetic datasets
A.I. Details of the datasets
Figure 1 shows evaluations of all selected models all benchmark datasets using MAE, Intra MI, Avg Inter MI, and Max Inter MI scores.
We provide key attributes of all datasets employed in this study in Table [ The datasets include seven benchmark datasets (ETThl, ETTh2, ETTm1, ETTm2, Weather, Traffic, and Electricity), eight synthetic dataset, and two real-world healthcare datasets (TIHM and MINDER). The benchmark datasets are publicly available on GitHub (https:: Semee-Seris-- Library/tree/maaattt /main). The health- care dataset TIHM is available on Zenodo (https://
Table 2 shows evaluations of variants of iTransformer model on all benchmark datasets using MAE, Intra MI, Avg Inter MI, and Max Inter MI scores. Table 3 gives the same evaluation on synthetic datasets. Tables 4 to pro- vide MAE and MSE of selected models benchmark and synthetic datasets for the Z-normalization ablation study.
www com/thuml/

---

A Closer Look at Transformers for Time Series Forecasting: Understanding Why They Work and Where They Struggle
Table I: Key attributes of the datasets: Dim represents the number of variates, Dataset Size indicates the total time points across (Train, Validation, Test) splits, Prediction Length specifies the number of future time points to predict (with four settings per dataset), and Frequency denotes the sampling interval of the time points.
Dataset
Dim
Prediction Length
Dataset Size
Frequency
Information
ETThI, ETTh2
7
(96. 192, 336, 720}
(8545, 2881, 2881)
Hourly
Electricity
ETTml, ETTm2
7
(96. 192, 336, 720}
(34465, 11521, 11521)
15 min
Electricity
Weather
21
(96, 192, 336, 720)
(36792, 5271, 10540)
10 min
Weather
Traffic
862
(96, 192, 336, 720)
(12185, 1757,3509)
Hourly
Transportation
ECL
321
(96, 192, 336, 720)
(18317, 2633, 5261)
Hourly
Electricity
Synthetic (8 subsets)
2
(96. 192, 336, 720}
(6132, 876, 1752)
Hourly
Synthetic
TIHM
8
{48, 96, 192]
(1496, 427, 213)
Hourly
Hcalthcare
MINDER
8
{48, 96, 192] (12281, 3508, 1754)
Hourly
Healtheare
Table 2: Results of variants of i'Trnffrmer model on benchmark datasets. "w/o SC" refers to the model without skip connections in the encoder layers, "VD-De" denotes the use of a variate-dependen decoder, "w/o SC & VVD-De" indicates both modifications applied, and "Original" represents the unmodified iTrrnsffommer model.
Pred
MAE
MSE
Dalasel
length
wlo SC
VD-De
w/o SC & VD-De
Original
w/o SC
VD-De
w/o SC & VD-De
Original
96
0,235 J. 0.001
0,218 1. 0.0
0,226 1: 0.002
0,214 J. 0.001
0.193 1: 0.001
0.174 1: 0.0
0,181 1: 0.001
0,175 1 0.001
192
0,277 + 0.001
0.259 - 0.001
0,265 + 0.003
0,257 + 0.001
0,245 + 0.001
0.222 = 0.001
0.229 + 0.003
0,224 + 0.001
Weather
336
0.311
0.001
0,3 + 0.001
0.303 +: 0.0
0,299 0.001
0.294 + 0.001
0.278 = 0.001
0,283 + 0.001
0,281 + 0.001
720
0.357
0.001
0.351 1: 0.002
0.352 1:: 0.001
0.35 1: 0.0
0.369 1 0.001
0.355 0.001
0.358 d: 0.002
0.36 0.001
96
0.433 = 0.0
0,427 + 0.004
0,467 + 0.015
0.409 - 0.0
0.429 + 0.001
0.412 = 0.006
0,478 + 0.025
0.394 + 0.001
192
0.461
0.453 + 0.002
0.014
= 0.001
0.481 + 0.002
0.461 0.003
0.528 =: 0.027
0.447 +: 0.001
E
336
0.482 0.001
0.471 1: 0.003
0.513 1: 0.009
0.464 0.001
0.524 1: 0.001
0.5 1: 0.003
0.565 1: 0.014
0.49 +: 0.001
720
0.51 + 0.002
0,499 = 0.002
0,536 + 0.002
0.497 + 0.001
0.539 + 0.005
0,52 + 0.004
0.575 + 0.005
0,511 + 0.002
96
0,366 + 0.001
0.361 + 0.003
0.371 + 0.002
0.35 + 0.001
0.317 + 0.001
0.312 = 0.005
0,321 + 0.003
0.3 : 0.001
192
0.412 1: 0.001
0.410 4: 0.002
0.416 1: 0.002
0.399 1: 0.0
0.397 1: 0.001
0.396 =: 0.003
0.401 +: 0.003
0.379 1:: 0.0
ETTb2
336
0.445 + 0.002
0.444 + 0.003
0.447 = 0.002
0.433 + 0.001
0.437 + 0.003
0.441 0.006
0.44 + 0.004
0.423 + 0.002
720
0.455
0.001
0.452 + 0.002
0.457 + 0.002
0.447 0.001
0.438 + 0.001
0.437 = 0.004
0.441 + 0.004
0.43 + 0.003
96
0.392
0.006
0.379 + 0.001
0.391 + 0.002
0.378 0.001
0.37 0.008
0.345 = 0.001
0.365 + 0.004
0.345 + 0.003
192
0.406 0.009
0.397 = 0.001
0.409 + 0.003
0.395 = 0.0
0.402 + 0.012
0.384 = 0.001
0.402 = 0.005
0.383 + 0.001
ETTml
336
0.426
0.004
0.42
0.001
0.418
0.434 1 0.005
0.424
0.439 = 0.004
0.418 1 0.002
720
0.46 1: 0.005
0.457 4: 0.001
0.467 1: 0.003
0.457 0.002
0.497 1: 0.007
0.492
0.004
0.509 1 0.006
0.49 +:: 0.003
96
0.279
0.002
0.275 1 0.0
0,282 1: 0.001
0.271 0.0
0.193 1. 0.001
0.188
0.001
0.196 1: 0.002
0,185 J. 0.001
192
0.317
0.002
0.315 + 0.001
0 + 0.001
0,311 0.001
0.256 +: 0.001
0.253 = 0.002
0.258 + 0.001
0.25 + 0.001
ETTm2
336
0.356 1: 0.003
0.353 4: 0.002
0.356 1 0.001
0.352 1:: 0.001
0.32 4: 0.005
0.316 + 0.002
0.321 :: 0.002
0.316 1: 0.001
720
0.410 + 0.001
0.408 + 0.002
0.41 = 0.002
0.406 + 0.001
0.418 + 0.001
0.415 = 0.003
0.42 - 0.002
0.412 + 0.002
96
0.287 + 0.004
0,240 + 0.0
0.35 + 0.0
0.24 + 0.0
0.9 + 0. 0.005
0. + 0.0
0.269 + 0.0
0.148 + 0.0
192
0.306 1: 0.004
0.259 _ 0.0
0.454 1 0.001
0.256 1: 0.0
0.211 1 0.005
0.165 + 0.0
0.271 1: 0.001
0.165 1 0.0
Electricity
336
0.325
0.003
0.274 + 0.00S
0.362 + 0.001
0.27 = 0.001
0.232 + 0.004
0.177 = 0.002
0,280 + 0.001
0.178 + 0.001
720
0.360
0.0066
0.3 +: 0.001
0.384
0.002
0.283
0.01
0.206
0.001
0.311 + 0.002
0.21 +: 0.001
96
0.663
0.005
0.276 + 0.001
0.799
0.005
0.268
1.123 + 0.016
0.396
0.001
1.406 + 0.002
0.392 + 0.001
192
0.559
0.004
0.284 1: 0.001
0.567 J. 0.006
0.278 0.001
0.945 1. 0.008
0.415
0.002
1.016 1 0.004
0.413 J. 0.002
Traflie
336
0.572
0.006
0.291 +: 0.002
0.581
+: 0.003
0.284 +: 0.0
1.004 0.01
0.427
0.001
1.029 + 0.001
0.426 + 0.0
720
0.569 1: 0.008
0.307 +: 0.001
0.571 +: 0.004
0.3 4: 0.001
0.996 4: 0.007
0.461 = 0.001
1.03 4: 0.001
0.458 4: 0.001
12

---

A Closer Look at Transformers for Time Series Forecasting: Understanding Why They Work and Where They Struggle
MAE
Intra
Avg. Inter MI
Max Inter
04
0.01
Elatricity
3-
0.3
0.1
Temoteer
Zonstomen
contooneer
Cresttement
Deenstument
Temeder
Pechtay
Temeter
Ansotommnt
rectormen
Crusttmmen
Palcarty
Tomeler
MAE
Avg.
Inter
15
2.0
0.10
0.5
0.05
0.0
(Bannstormtt
0.
Auteborner
Crensformer
Itaestormnt
Treeter
0.09
Timatee
24
Zanstomen
Autotommnt
Crevttement
Trmater
MAE
Intra MI
Avg. Inter
Max Inter MI
0.35
0.09
5.2
Autorammnn
Patcersy
mecatmme
Inter
2.
0.15
8.4
22-
02
0.0
34
0.00
Ceensfomme
Timatter
Crosetmmer
Patlansy
(ranstoment
ranstomorr
Antettrmnt
Chasstonnr
Treattr
Antotommn
Prcsumm
Crussdemes
Towster
MAE
Intra M)
Avg. Inter
Max Inter Mi
0.8
5:100
5.055
02
3.000
Pansk
Temmuder
renforme
Toseder
Vemude
Crasterten
mevnstomen
Temeudor
MAE
Intra Ml
Avg. Inter MI
Max Inter Mi
traffic
0.005
0.20
0.004
0.05
0,000
PORDTTS
Towederr
Punstomerr
Cooatumaa
Peschrey
MAE
Intra
Avg. Inter MI
Max Inter MI
1.0
6.255
weatthe
0.200
5.225
Covaterme
Towwker
Aundemter
Reausormet
Tomater
recoooeeer
Croutenserr
Timassor
Rnserdottor
Prediction length
96
192
336
720
Figure Performance comparison of selected transformer- based models on commonly used benchmark datasets.
13

---

A Closer Look at Transformers for Time Series Forecasting: Understanding Why They Work and Where They Struggle
Table 3: Results of variants of iTrrnformer model on synthetic datasets. "w/o SC" refers to the model without skip connections in the encoder layers, "VD-De" denotes the use of variate- drrendeddeeeneent decoder, "w/o SC & VD-De" indicates both modifications applied, and "Original" represents the unmodified iTTransformer model.
Pred
MAF
MSF
Dataset
longth
w/o SC
VD-De
w/o SC & VD-De
Original
w/o SC
VD-De
wlo SC & VD-De
Original
96
0.509 = 0.071
0.422 + 0.0
0.42 + 0.0
0.42 = 0.0
0.426 = 0.12
0.28 + 0.0
0.278 = 0.0
0.277 + 0.0
Synthetic
192
0.421 + 0.001
0.419 +: 0.0
0.419 + 0.0
0.418 + 0.0
0,279 = 0.001
0.277 0.0
0.275 = 0.0
0,275 + 0.0
= 0.95
336
0.420 + 0.0
0.418 + 0.0
0:418 = 0.0
0.417 + 0.0
0.277 +: 0.0
0.274 0.0
0.274 0.0
0.273 = 0.001
0.0
0.418 1: 0.0
0.419 +: 0.0
0.277 1: 0.0
0.273
0.273
0.272 1: 0.001
0.002
0.43
0,429 + 0.0
0.322 0.004
0.289
0.287
0.288 + 0.0
Syuthetic
192
0.431 1. 0.0
0.426 1: 0.0
0.426 1: 0.0
0.426 1: 0.0
0.289 1: 0.0
0.283 0.0
0,283 1: 0.0
0.283 1: 0.001
0.95
336
0.43 + 0.0
0.426 +: 0.0
0.426 +: 0.0
0.426 0.001
0.29 0.9
0.284 0.0
0.284 :: 0.0
0.284 4: 0.001
0.2
720
0.432 + 0.0
0.428 +: 0.0
0.429 = 0.0
0.429 + 0.0
0.292 + 0.0
0.287 0.0
0.287 - 0.0
0,287 + 0.0
96
0.436 1: 0.001
0.432 J: 0.001
0.431 1: 0.0
0.43 1: 0.0
0.299 =: 0.001
0.292 1: 0.001
0.29 1: 0.0
0.29 1: 0.001
Synthetic
192
0.435 + 0.0
0.432 + 0.0
0.429 = 0.0
0.43 + 0.001
0.296 + 0.0
0.292 + 0.0
0.289 = 0.0
0.29 + 0.001
0.95
336
0.432 +: 0.0
0.427 +: 0.0
0.428 +: 0.0
0.428 +: 0.001
0.295 +: 0.0
0.287 + 0.0
0.289 -: 0.0
0.289.1 0.001
a 0.4
720
0.435 0.001
0.429 + 0.0
0.432 = 0.0
0.43 = 0.0
0.299 = 0.001
0.29 + 0.0
0.293 0.0
0.292 + 0.0
0.444 0.001
0,447 + 0.002
0.435 + 0.0
0.448 0.001
0.308
0.001
0.312 0.003
0.297 0.0
0.317 + 0.002
Synthetic
U.456 + 0.002
0.0
0.322
0.001
0.33 0.003
0.325 + 0.002
0.95
336
0.454
0.001
0.462 1: 0.005
0.45 1: 0.001
0.455 0.003
0.326
0.001
0.337 0.008
0.319 1: 0.001
0.328 1: 0.004
0.8
720
0.469
0.001
0.48
0.002
0.466 1: 0.001
0.469 0.002
0.348 + 0.001
0.366 1: 0.004
0.343 4: 0.001
0.348 1: 0.003
96
0.791 + 0.0
0.789 + 0.001
0.79 0.0
0,792 + 0.0
0.975 + 0.0
0.972 + 0.001
0.973 + 0.0
0.977 + 0.0
Synthetic
192
0.79 1: 0.001
0.79 1: 0.0
0.79 1: 0.0
0.79 =: 0.0
0.973 1: 0.0
0.974 1: 0.001
0.973 :: 0.0
0.973 J: 0.0
0.5
336
0.788 = 0.001
0.788 + 0.001
0.788 = 0.0
0.788 + 0.0
0.968 + 0.0
0.968 + 0.001
0.968 = 0.0
0.967 + 0.0
0
720
0.786 + 0.0
0.784 +: 0.001
0.785 + 0.0
0.786 +: 0.001
0.963 = 0.001
0.96 1: 0.001
0.961 - 0.0
0.964 + 0.001
96
0.786 1: 0.0
0.786 1: 0.0
0.787 1: 0.0
0.787 1: 0.0
0.975 1: 0.0
0.972 1: 0.001
0.975 1: 0.0
0.976 1:: 0.0
Synthetic
192
0.788 + 0.0
0.787 + 0.0
0.788 = 0.0
0.788 0.0
0.977 + 0.0
0.971
0.977 0.0
0.977 + 0.0
7 0.5
0.788 +:
0.789 + 0.0
0.977 +
0.972 0.001
0.977
+: 0.001
0.2
720
0.791
0.791 = 0.001
0.791 = 0.0
0.981 0.002
0.979 0.002
0.981
0.981 = 0.0
96
0.802 + 0.0
0.798 = 0.001
0.802 0.0
0.8
1.003 0.001
0.994 0.003
1.004 = 0.0
0.999 + 0.001
Synthetic
192
0.799 + 0.0
0.794 + 0.0
0.799 + 0.0
0.798 # 0.0
0.999 = 0.001
0.986 + 0.001
0.998 = 0.0
0.996 + 0.0
0.5
336
0.798 1: 0.0
0.795 1: 0.001
0.798 :: 0.0
0.797 1: 0.0
0.994.1: 0.0
0.986 +: 0.002
0.994 :: 0.0
0.993 d: 0.0
0.4
720
0.795 1: 0.001
0.794 1: 0.001
0.795 1: 0.0
0.795 1: 0.0
0.986 :: 0.001
0.984 1: 0.001
0.985 :: 0.0
0.985 1: 0.0
96
0.761 + 0.0
0.757 + 0.001
0.76 + 0.001
0,761 + 0.0
0.91 + 0.001
0.9 + 0.001
0.908 + 0.003
0.911 +: 0.001
Synthetic
192
0.758 1 0.0
0.755 1: 0.001
0.757 1: 0.001
0.759 1: 0.001
0.905 :: 0.001
0.898 1:: 0.001
0.904 1: 0.002
0.907 1: 0.0
7 0.5
336
0.765 0.001
0.762 + 0.001
0.763 + 0.001
0.766 0.001
0.921 = 0.003
0.915
0.002
0.918 + 0.002
0.925 - 0.004
0.8
0.001
+: 0.003
0.001
0.936 0.003
0.935
0.936 4:
0.002
14

---

A Closer Look at Transformers for Time Series Forecasting: Understanding Why They Work and Where They Struggle
Table 4: The effect of Z-normalization MAE of various models across benchmark datasets.
Metric
Model
7-Norm
Weather
FT
ETTh2
ETTml
FTTm2
Electricity
Traffic
length
0
0.262 0.006
192
0.247 0.001
0.422 0.003
0.409 1: 0.004
0.385 0.001
0.308 0.003
0.251 + 0.0
0.273 + 0.003
w/
336
0.292 = 0.001
0.45 = 0.003
0.459 + 0.011
0.410 0.003
0.349
0.003
0.269 = 0.002
0.294 + 0.003
720
0.345 + 0.002
0,502 0.007
0.467 + 0.002
0.452 0.001
0.408
0.001
0.303 + 0.005
0.321 + 0.009
Crossformer
96
0.226 = 0.01
0.423 0.012
0.592 +: 0.048
0.409 = 0.024
0,392 + 0.03
0.248 +: 0.001
0.27 + 0.008
192
0.277 1: 0.007
0.459 0.022
0.897 1: 0.203
0.505 +: 0.046
0.575
0.024
0.269 1: 0.011
0.284 1: 0.004
wio
336
0.328 = 0.006
0.047
1.04 0.114
0.609 = 0.025
0,766 0.226
0.289 + 0.007
0.299 0.005
0.412
1.458 0.216
0.66
0.003
0.215 0.001
0.345 0.004
0.368 0.007
0.261
0.276 0.001
0.298
192
0.257 0.0
0.431 0.001
0.399 1. 0.002
0.387 1: 0.004
0.307
0.003
0.283 1: 0.0
0.303 1 0.001
w/
336
0,297 + 0.0
0.460
0.004
0.438 + 0.004
0.409 = 0.001
0,349
0.001
0,298 + 0.001
0.308 + 0.001
720
0.346 + 0.0
0.5 + 0.002
0.456 + 0.003
0.443 = 0.001
0.405
0.004
0,331 + 0.0
0.326 + 0.001
PatcTTST
96
0.234 1: 0.002
0.417
0.003
0.392 1. 0.034
0.399 1: 0.004
0.305
0.005
0.28
0.001
0.306 1: 0.001
192
0.273 + 0.003
0.451 0.012
0.472 1: 0.012
0.417 = 0.004
0.391
0.056
0.286 1: 0.001
0.308 0.0
w/o
0.316 +
0.315
0.378 0.013
0.698 0.032
0.478 0.003
0.008
0.332
0.334 0.001
MAE
96
0.214 1: 0.001
0.409 0.0
0.35 0.001
0.378 :: 0.001
0.271 0.0
0.24 0.0
0.268 1: 0.0
192
0.257 = 001
0.440
0.001
0.399 - 0.0
0.395 + 0.0
0 + 0.001
0.256 =: 0.0
0.278 + 0.001
W
336
0.299 1: 0.001
0.464
0.001
0.433 1: 0.001
0.418 1 0.0
0.352 0.001
0.27 1: 0.001
0.284 1: 0.0
720
0.35 = 0.0
0.497 0.001
0.447 + 0.001
0.457 = 0.002
0.406 0.001
0.299 + 0.001
0.3 + 0.001
0.226 + 0.001
0.433
0.002
0.492 1: 0.022
0.409 0.004
0.331 0.011
0.252 + 0.001
0.306 + 0.002
0.437
0.001
0.305
wif
336
0.32 0.002
0.512 0.004
0.713 0.039
0.471 0.007
0.613 0.056
0.288 = 0.0
0.315 + 0.008
720
0.37 0.004
0.565 0.003
0.801 +: 0.027
0.509 0.009
0.886 0.041
0.322 + 0.006
0.334 + 0.005
96
0.206 + 0.0
0.404 0.001
0.34 =: 0.002
0.36 0.002
0,255 0.001
0.243 + 0.002
0.274 + 0.001
192
0.249 1: 0.001
0.439
0.002
0.394 1. 0.006
0.385 1: 0.001
0,301 1: 0.0
0.256 1. 0.001
0.283 1L 0.0
w/
336
0.291 + 0.001
0. 0.462 + 0.01
0.433 + 0.005
0.407 + 0.0
0.34 = 0.003
0.274 + 0.002
0.289 + 0.001
720
0.343 + 0.001
0.487
0.018
0.447 1: 0.003
0.441 + 0.001
0.397 0.002
0.503 + 0.005
0.308 + 0.001
TimeXer
0.232 + 0.002
0.438 0.005
0.756 0.078
0.420 = 0.003
0.412 0.013
0.252 + 0.002
0.32 0.003
0.277 0.002
0.027
0.445
0.27
0.321 0.003
w/f
336
0.334 + 0.007
0.489
0.009
0.906 1: 0.016
0.499 0.011
0.683 0.082
0.293 + 0.001
0.322 + 0.002
720
0.399 + 0.002
0.569 0.007
1.036 + 0.054
0.528 = 0.012
1.219 0.067
0.33 + 0.004
0.343 + 0.004
15

---

A Closer Look at Transformers for Time Series Forecasting: Understanding Why They Work and Where They Struggle
Table 5: The effect of Z-normalization MSE of various models across benchmark datasets.
Metric
Model
7-Norm
Weather
FT
ETTh2
ETTml
FTTm2
Electricity
Traffic
length
0.154
0.421 0.004
192
0.203 0.001
0.44 + 0.001
0.396 1: 0.006
0.367 0.0
0.248 0.002
0.158 0.001
0.44 4: 0.006
w/
336
0.265 + 0.002
0.472 0.002
0.456 + 0.014
0.410 0.006
0.312 + 0.005
0.175 = 0.002
0.477 = 0.007
720
0.343 = 0.003
0,519 0.009
0.46 = 0.003
0.482 0.005
0,413 + 0.002
0.214 + 0.006
0.546 + 0.011
Crossformer
96
0.151 + 0.005
0.398 0.009
0.715 + 0.132
0.373 = 0.025
0.328
0.063
0.147 : 0.001
0.522 + 0.021
192
0.203 +: 0.002
0.454 +: 0.02
1.435 1: 0.414
0.486 +: 0.063
0.655 0.071
0.169 1: 0.007
0.55 0.009
wio
336
0.262 + 0.005
0.621
0.077
1.82 0.344
0.677 = 0.045
1.203
0.197 + 0.009
0.588 0.027
0.407 0.027
0.758
0.604 0.007
0.173 0.001
0.292 0.005
0.329
0.184 0.0
0.459 0.001
192
0.221 1 0.001
0.431
0.002
0.382 1. 0.007
0.366 0.005
0.246
0.002
0.191 1: 0.0
0.468 1: 0.001
w/
336
0,276 +: 0.0
0.482 = 0.013
0,428 + 0.005
0.397 0.001
0,311
0.001
0.207 = 0.0
0.483 + 0.001
720
0.353 + 0.0
0.519
0.002
0.442 + 0.006
0.457 = 0.001
0.41 = 0.007
0.249 +: 0.001
0.517 + 0.002
PatcTTST
0.169 1: 0.001
0.396
0.004
0.364 1. 0.052
0.369 1: 0.002
0.215 1L 0.01
0.184 1: 0.001
0.569 1: 0.001
192
0.21 0.001
0.446 +: 0.01
0.488 1: 0.016
0.399 = 0.005
0.344
0.072
0.19 0.001
0.581 0.0
w/o
0.262 + 0.01
+ 0.004
+
0.597
0.023
0.972 0.104
0.489 0.004
0.241
0.632 0.001
MSE
96
0.175 1 0.001
0.394
0.001
0.3 0.001
0.345 :: 0.003
0,185 0.001
0.148 1: 0.0
0.43 0.003
192
0.224 = 0.001
0.447 = 0.001
0.379 + 0.0
0.383 = 0.001
0.25 0.001
0.166 = 0.0
0.452 = 0.003
w//
336
0.281 1: 0.001
0.490
0.001
0.423 1: 0.002
0.418 1: 0.002
0.316 0.001
0.178 1:: 0.001
0.473 +:: 0.001
720
0.36 :: 0.001
0,511
0.002
0.43 + 0.003
0.49 + 0.003
0,412 + 0.002
0.21 + 0.001
0.523 + 0.005
0.166 + 0.001
0.418
0.001
0.497 1: 0.028
0.374 = 0.005
0.241 0.011
0.153 + 0.0
0.54 0.002
0.217
0.420
0.553
wif
336
0.274 0.004
0.542
0.939 0.078
0.469 0.009
0.695 0.127
0.185 0.001
0.58 0.016
720
0.344 + 0.005
0.596 0.003
1.124 + 0.061
0.537 0.01
1.273
0.125
0.221 0.006
0.612 + 0.044
96
0.158 + 0.0
0,385
0.002
0.288 + 0.003
0.324 = 0.003
0.17 + 0.0
0.141 + 0.001
0.43 0.003
192
0.206 1: 0.001
0.435
0.003
0.373 1. 0.011
0.364 1: 0.001
0,239
0.001
0.158 1: 0.0
0.452 1: 0.003
w/
336
0.263 + 0.001
0.492
0.013
0.421 + 0.005
0.396 = 0.001
0.299
0.003
0.176 # 0.002
0.473 + 0.001
720
0.342 + 0.001
0.506 + 0.03
0.433 1: 0.003
0.453 = 0.002
0.397
0.003
0.207 + 0.007
0.523 + 0.00S
TimeXer
0.163 0.002
= 0.004
1.132 0.152
0.390 0.006
0.370
0.027
0.15 0.001
0.573 + 0.003
0.211 0.001
1.559 0.089
0.425
0.17
0.595
w/f
336
0.274 + 0.004
0.504 0.009
1.618 + 0.084
0.502 0.014
0.999 0.213
0.194 1 0.002
0.604 + 0.004
720
0.379 + 0.006
0.598 0.014
1.912 = 0.14
0.551 = 0.018
2.616 0.208
0.235 # 0.004
0.643 + 0.006
16

---

A Closer Look at Transformers for Time Series Forecasting: Understanding Why They Work and Where They Struggle
Table 6: The effect of Z-normalization on MAE of various models across synthetic datasets.
Pred
Synthetic (y 0.95)
Synthetic (? U.5)
Metric
Model
Z-Norm
length
0.2
0.4
0.8
a==
0.2
0.4
0.8
96
0.424
0.434
0.434 = 0.0
0.436 0.0
0.789
0.787 0.0
0.801 + 0.0
0.762 + 0.001
192
0.423 + 0.0
0,431 0.0
0.433 # 0.0
0.449 0.0
0.789 0.0
0.788 + 0.0
0.798 + 0.0
0.759 = 0.0
w/
336
0.423 +: 0.0
0.431 +: 0.0
0.432 + 0.0
0.451 +: 0.001
0.786 0.0
0.789 +: 0.0
0.797 +: 0.0
0.767 4: 0.002
720
0.425 + 0.0
0.435 = 0.0
0.435 = 0.0
0.472 = 0.004
0.783 0.0
0.79 + 0.0
0.794 + 0.0
0.772 + 0.001
Crossformer
0.419 0.0
0.427 0.0
0.427 + 0.0
0.421 0.0
0.782 0.0
0.784 +: 0.0
0.794 + 0.0
0.748 + 0.001
192
0.418 0.0
0.426 0.0
0.427 = 0.0
0.427 0.0
0.781 = 0.0
0.783 + 0.0
0.792 + 0.0
0.752 = 0.0
wif
0.418 0.001
0.426
0.426 1: 0.001
1:
1. 0.0
0.749
720
0.42 + 0.0
0.428
0,429 = 0.0
0.433 +
0.774
0.785 0.0
0.787 + 0.0
0.747 0.0
96
0.424 + 0.0
0,433 0.0
0.433 # 0.0
0.438 + 0.0
0.794 0.0
0.789 + 0.0
0.8 + 0.0
0.769 + 0.001
192
0.422 1: 0.0
0.43 1: 0.0
0.433 1: 0.0
0.452 1 0.0
0.793 0.0
0.791 1: 0.0
0.797 J. 0.0
0.763 0.0
w/
336
0.421 + 0.0
0.43 + 0.0
0.431 + 0.0
0.452 + 0.0
0.789 0.0
0.79 +: 0.0
0.797 1: 0.0
0.775 1 0.006
720
0.421 + 0.0
0.432 + 0.0
0.434 + 0.0
0.47 + 0.0
0.786 0.001
0.791 + 0.0
0.794 + 0.0
0.782 + 0.008
PatcTTS
0.418 0.0
0.425 0.0
0.424 = 0.0
0.422 0.0
0.787 = 0.0
0.787 + 0.0
0.794 + 0.0
0.757 + 0.001
0.424
0.425 1
0.426
0.785 1:
0.793 1: 0.0
0.756
wifa
336
0.416
0.424 0.0
0.424 0.0
0.429
0.782 0.001
0.783 0.0
0.791 + 0.0
0.756 0.0
720
0.416 0.0
0.425 0.0
0.427 1: 0.0
0.434 0.0
0.777 0.0
0.786 1: 0.0
0.788 1: 0.0
0.753 1: 0.001
MAE
96
0.42 + 0.0
0.43 + 0.0
0.43 0.0
0.448 = 0.001
0.792 0.0
0.787 + 0.0
0.8 + 0.0
0.761 0.0
192
0.418 + 0.0
0.426 1: 0.0
0.43 + 0.001
0.452 : 0.001
0.79 + 0.0
0.788 + 0.0
0.798 : 0.0
0.759 = 0.0
336
0.417 + 0.001
0.426 0.001
0.428 1: 0.001
0.455 + 0.003
0.788 0.0
0.789 + 0.0
0.797 1: 0.0
0.766 1: 0.001
720
0.418
0.429 0.0
0.43
0,469 0.002
0,786
0.001
0.791 + 0.0
0.795 + 0.0
0.772 + 0.001
iTransformer
0.423 + 0.0
0.429 0.004
0.785 + 0.0
0.794 + 0.0
0.752
0.424
0.793 + 0.0
0.752
w/o
336
0.413 1 0.001
0.422 0.001
0.422 1 0.0
0.429 0.001
0.781 0.0
0.782 1 0.0
0.791 1. 0.0
0.75 0.001
720
0.413 + 0.0
0.423 0.0
0.424 = 0.0
0.427 + 0.0
0.776 0.0
0.786 + 0.0
0.788 + 0.0
0.749 = 0.0
96
0.423 + 0.0
0.433 +:: 0.0
0.433 + 0.0
0.442 + 0.001
0.79 + 0.0
0.786 + 0.0
0.798 + 0.0
0.76 0.001
192
0.421 + 0.0
0.43 + 0.0
0.434 + 0.0
0.452 + 0.0
0.79 + 0.0
0.788 + 0.0
0.796 + 0.0
0.759 + 0.002
336
0.42 1: 0.0
0.43 1: 0.0
0.432 : 0.0
0.455 0.0
0.788 0.0
0.788 1 0.0
0.795 1: 0.0
0,766 1: 0.001
0.435 + 0.0
0.471
0.79
0.793 4 0.0
0.773
TimeXer
0.417
0.426
0.425 0.0
0.425 0.001
0.784 0.0
0.792 - 0.0
0.748 0.0
192
0.415
0.424
0,426 0.0
0.428
0.782
0.782 0.0
0.791 - 0.0
0.751 + 0.001
w/o
336
0.415 0.0
0.424 0.0
0.424 + 0.0
0.43 + 0.0
0.78 + 0.0
0.781 + 0.0
0.789 + 0.0
0.75 0.0
720
0.416 +: 0.0
0.424 1: 0.0
0.427 :: 0.0
0.434 4: 0.0
0.776 0.0
0.785 1: 0.0
0.786 4: 0.0
0.749 =: 0.0
17

---

A Closer Look at Transformers for Time Series Forecasting: Understanding Why They Work and Where They Struggle
Table 7: The effect of Z-normalization on MSE of various models across synthetic datasets.
Pred
Synthetic (y 0.95)
Synihetic (y 0.5)
Metric
Model
Z-Norm
length
0.2
0.4
0.8
a==
0.2
0.4
0.8
96
0.283
0.293 0.0
0.296 = 0.0
0.297 0.0
0.972
0.974 + 0.0
1.0
0,912 0.002
192
0.281 + 0.0
0.29 +: 0.0
0.295 # 0.0
0.316 + 0.001
0.97 + 0.0
0.975 +: 0.0
0.995 0.0
0,907 0.001
w/
336
0.28 1: 0.001
0.291 1: 0.0
0.295 + 0.0
0.319 + 0.001
0.964 0.0
0.975 4: 0.0
0.991 +: 0.0
0,928 1: 0.006
720
0.281 = 0.001
0.295 = 0.0
0.299 = 0.0
0.351 = 0.005
0.957 = 0.001
0.978 + 0.0
0.983 = 0.001
0.941 + 0.003
Crossformer
96
0.276 0.0
0.284 0.0
0,288 + 0.0
0.279 0.0
0.953 = 0.0
0.965 + 0.0
0.984 0.0
0,878 0.002
192
0.274 0.0
0.284 0.0
0,288 + 0.0
0.287 0.0
0.949 0.0
0.96
0.982 0.0
0.889 = 0.0
wif
0.274 0.001
0.284
0.287 1: 0.001
0.289
0.957 1.
0,883
720
0.275
0.287
0.29
0.297
0.937
0.963 + 0.0
0.966 0.001
0.881
96
0.283 + 0.0
0.292 0.0
0.293 + 0.0
0.3 = 0.0
0.982 0.0
0.981 + 0.0
0.999 0.0
0.929 0.002
192
0.281 1: 0.0
0.289 1: 0.0
0.294 =: 0.0
0.32 1: 0.001
0.977 0.0
0.983 1. 0.0
0.995 1: 0.0
0,918 0.001
w/
336
0.279 + 0.0
0.29 +: 0.0
0,293 + 0.0
0.32 1: 0.0
0.97 : 0.001
0.98 +: 0.001
0.99 1 0.001
0,948 0.016
720
0.276 + 0.0
0.292 + 0.0
0.297 + 0.0
0.347 + 0.001
0.963 0.001
0.981 + 0.001
0.982 + 0.001
0.966 0.021
PatcTTS
0.275 0.0
0.281 0.0
0,282 + 0.0
0.28 + 0.0
0.962 = 0.0
0.972 + 0.0
0.984 0.0
0.001
0.273
0.284 1: 0.0
0.287
0.967 1 0.0
0.983
wifa
336
0.273
0.281 0.0
0.283 0.0
0.29 0.0
0.951 0.002
0.962 +:2 0.001
0.977 0.001
0.001
720
0.271 0.0
0.282 0.0
0.286 1: 0.0
0.297 0.0
0.944
0.001
0.967 1: 0.001
0.967 0.001
0.897 0.002
MSE
96
0,277 + 0.0
0.288 0.0
0.29 0.0
0.317 = 0.002
0.977 0.0
0.976 + 0.0
0.999 = 0.001
0,911 0.001
192
0.275 + 0.0
0,283 +: 0.001
0.29 = 0.001
0.325 + 0.002
0.973 0.0
0.977 + 0.0
0.996 + 0.0
0.907 1: 0.0
336
0.273 + 0.001
0.284 0.001
0.289 1 0.001
0.328 + 0.004
0.967 = 0.0
0.977 +: 0.001
0.993 11: 0.0
0.925 0.004
720
0.272 0.0
0.287 0.0
0.292 - 0.0
0.348 0.003
0.964
0.001
0,981 + 0.0
0.985 0.0
0.939 0.002
iTransformer
0.281 + 0.0
0.291 0.006
0.0
0.967 + 0.0
0.983
+ 0.001
192
0.273
0.283 + 0.001
0.295
0.962 +: 0.0
0.983
w/o
336
0.273 1 0.0
0.279 0.001
0.282 1. 0.001
0.292 0.001
0.949 0.0
0.96 0.001
0.978
0.886 0.001
720
0.271 + 0.0
0.279 # 0.0
0.283 + 0.0
0.29 + 0.0
0.942 0.0
0.966 + 0.0
0.968 0.0
0.886 0.0
96
0.282 + 0.0
0.293 ::: 0.0
0.294 + 0.0
0.307 + 0.001
0.973 0.0
0.974 + 0.0
0.996 + 0.001
0.908 1: 0.002
192
0.28 + 0.0
0.289 + 0.0
0.295 + 0.0
0.323 + 0.001
0.972 0.0
0.976 + 0.0
0.991 = 0.001
0.907 + 0.005
336
0.278 0.0
0.29 1: 0.0
0.294 :: 0.0
0.327 0.0
0.966 0.0
0.974 1: 0.0
0.988 0.001
0.926
0.004
0.297 + 0.001
0.35 0.003
0.0
0.977 + 0.0
0.982 0.001
TimeXer
0.275
0.282
0.284 + 0.0
0.285 0.002
0.965 + 0.0
0.978
- 0.0
192
0.273
0.281
0.285 0.0
0.289
0.951
0.96
0.978
0.887 0.004
w/o
336
0.273 0.0
0.282 0.0
0.284 + 0.0
0.292 0.0
0.948 0.0
0.956 + 0.0
0.972 =:: 0.0
0.885 # 0.0
720
0.271 4: 0.0
0.281 1: 0.0
0.286 :: 0.0
0.297 +: 0.0
0.941 0.0
0.963 1: 0.0
0.961 d: 0.0
0.887 1: 0.0
18


Reviews:
[{"review_id": "2M9vpTKAl8", "reviewer": "Reviewer_npeJ", "summary": "The authors survey the literature on time-series forecasting with transformers, and divide previously published approaches into 3 categories.  Given multiple time-varying signals (variates), the signals can be chopped into tokens along the time axis, along the variate axis (one token per variate), or both.  \n\nThe authors then analyze the performance of various transformer architectures on a variety of commonly used benchmark data sets.  They find that intra-variate dependencies in the data are much more predictive than inter-variate dependencies.  Point-wise models, which bundle all variates together into a single token, perform poorly because they fail to capture patterns within a single variate. \n\nThe authors also design a set of synthetic datasets, in which the inter-variate and intra-variate dependencies can be precisely controlled, to further back up their findings.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your encouraging comments. We truly appreciate your effort and time for reviewing our work. \n\n*Response to questions:*\n\n1. Why use transformers?\n\n    In previous work on developing new transformer architectures for time series forecasting, there has been little validation of whether the attention mechanism functions as intended. Although several studies aim to enhance transformers' ability to capture inter-variate dependencies, none have explicitly verified this in relation to the underlying data characteristics and model outcomes. This lack of validation motivated us to explore the issue in our study. \n\n    Our findings show that patch-wise and variate-wise transformers effectively capture intra-variate patterns, which remains a useful property in time series forecasting. Additionally, experiments with synthetic datasets demonstrate that certain transformer architectures—such as Crossformer—are capable of learning inter-variate patterns. This suggests that transformers can still be effective in scenarios where such patterns are important.  We hope that our findings can offer useful insights for designing new architectures or for more effectively applying transformers in related applications.\n\n2. Is attention useful in iTransformer?\n\n    As we understand it, attention is still useful in iTransformer. For example, in our skip-connection ablation study, we observed that iTransformer performed better without skip connections on synthetic datasets with high inter-variate dependencies. This suggests that, in such scenarios, the attention mechanism alone is still capable of capturing meaningful patterns. Additionally, the skip connection in iTransformer is added to the attention output—not as a simple bypass, but rather as a means to reinforce the self-attention for each variate.", "claims_and_evidence": "The authors do a good job on the literature survey, covering a variety of recently published architectures.  The experiments, using both benchmark datasets and custom synthetic datasets also seem well-designed.  The author's analysis is both thoughtful and detailed.", "methods_and_evaluation_criteria": "The methods and evaluations make sense.", "theoretical_claims": "The only theory was in the equations for mutual-information.  I did not find any errors, but this is not my area of expertise.", "experimental_designs_or_analyses": "The experiments seem to be well designed.", "relation_to_broader_scientific_literature": "The literature survey, which compares the different architectures of various models, was particularly well done.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "I do have one question.  I am not very familiar with time-series forecasting, but I was a bit confused after reading this paper.  The variate-wise models, like iTransformer, do not use attention to capture temporal (intra-variate) dependencies, and they still perform well.  But if attention is not being used to capture temporal dependencies, then why use a transformer for time-series modeling at all?  \n\nIn normal language modeling, the whole point of attention is that it's very good at capturing long-range temporal dependencies.  If attention is not good at capturing such dependencies in time series data, then perhaps the transformer is not an appropriate architecture.  The authors further  reinforce this idea by pointing out that the skip connection, which bypasses attention, is crucial to model performance.  Am I missing something?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MzMzq6YrZg", "reviewer": "Reviewer_iw3r", "summary": "There has been many proposals of transformer architectures for time series forecasting, and some of them are simpler some are more sophisticated, some work well, and some struggle. This work, examines why some of them work better and some struggle: In doing so, it uses an existing classification (Wang et al 2024), point wise, patch-wise and variate-wise tokens, and uses  a representative of each one, and introduce mutual information metric and additional synthetic datasets to test them. They show that intravariate dependencies are the primary source of contribution to prediction performance, while intervariate dependencies play rather a minor part.  Moreover,  Z-score normalisation and skip connections play a crucial role, in their ablation studies.  They validate these insights through synthetic and real world data sets.\n\n##UPDATE AFTER REBUTTAL##\n\nI maintain my initial positive score after the rebuttal, thanks to the quality of the work, and well-addressing of my questions by the authors.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your encouraging comments. We truly appreciate your effort and time for reviewing our work.\n\n*Response to Claims And Evidence:*\n\n1. In Table 3, removing the skip connection leads to a clear performance drop across all benchmark datasets, with particularly significant degradation on Electricity (MAE increases from 0.266 to 0.320) and Traffic (MAE increases from 0.283 to 0.591). On the synthetic datasets, this degradation becomes negligible while $\\alpha$ increases under the condition of $\\gamma$ = 0.95. In the manuscript, we stated that \"removing the skip connection notably degrades performance across most datasets, **except** for synthetic datasets with high inter-variate dependencies ($\\alpha$ = 0.8) or low autocorrelation ($\\gamma$ = 0.5),\" which aligns with your observation that \"low autocorrelation does not seem to be degraded at all.\" We will revise this statement to improve its clarity in the revision.\n\n2. Apologies for the confusion—this statement was incorrect. It should be: \"It improves performance on synthetic datasets with high inter-variate dependencies ($\\alpha$ = 0.8) under low autocorrelation ($\\gamma$ = 0.5).\" We will revise the corresponding claim to: \"The variate-dependent decoder has the potential to enhance the model’s ability to capture inter-variate interactions, particularly in scenarios with strong dependencies between variates.\"\n\n*Response to broader literature and essential references:*\n\n&nbsp;&nbsp;&nbsp;&nbsp; Thank you for your valuable suggestions. We will add discussions of both papers in the Related Work section.\n\n*Response to questions:*\n\n1. Why was TFT not included?\n\n    In this study, we focused on representative transformer architectures from the perspective of token representations within the attention mechanism. TFT, however, is a hybrid model that incorporates variable selection and LSTM-based encoders prior to applying attention. As a result, its token representations are less explicit and not as directly comparable to the models we selected. We agree that TFT is a relevant model and will include a discussion of it in the revised manuscript.\n\n2. What is the major role of MAX MI?\n\n    We propose MAX MI as a measure of the mutual information captured by a model between the most strongly interacting variates. This metric helps assess whether a model is effectively learning inter-variate dependencies, particularly in cases where the number of variates is large and only a few exhibit strong interactions. For instance, in Figure 3, Crossformer achieves the highest MAX MI on the Traffic dataset (which has 862 variates), while its average inter-variate mutual information (Avg. Inter MI) remains lower than that of most other models.\n\n3. Explanation of $\\sigma_{ij}$?\n\n    $\\sigma_{ij}$ estimates the extent to which changes in the prediction of variate $j$  are caused by changes in variate $i$. Unlike Pearson's correlation, it is more versatile as it captures both linear and non-linear relationships.  \n    Mathematically, $\\sigma_{ij}^2$ is the variance of the predictions of variate $j$ conditioned on inputs $\\mathbf{x}$ where all variates are held fixed except for variate $i$. To estimate this conditional variance, the variation of variate $i$ is introduced through N different samples by augmenting original samples in the dataset.  Specifically, each original sample is augmented into N = 5 versions, differing only in the value of variate $i$:  one instance is set to zero, one retains the original value, and the remaining instances are generated by adding Gaussian noise of varying strengths to the original value. We will amend the explanation in the revised manuscript to improve the clarity of the paper.", "claims_and_evidence": "In general claims and evidences are in line and are very well. There are some claims that are hard to justify in the presented tables: \n\nClaim 1: \"removing skip connection notably degrades the performance across most datasets, except for synthetic datasets with high inter- variate dependencies (α = 0.8) or low autocorrelation (γ = 0.5). \"\n\nThe difference seems to minuscule, or does not exist: so mainly on the ones with the no interrelation (alpha = 0).  Moreover low autocorrelation does not seem to be degraded at all. \n\n\nClaim 2:  \"Replacing variate independent decoder with with a variant dependent decoder  improves performance on synthetic datasets with high inter-variate dependencies (α = 0.8) or low auto- correlation (γ = 0.5).\"\n\nThe results are either so minuscule or non-existent that such bold claim that follows is difficult to make.", "methods_and_evaluation_criteria": "Definitely makes sense. In general I like the mutual information tests, and the ablation studies ideas. In order to do that paper tries to characterise the data we are testing these systems, and this looks like the obvious viable option.", "theoretical_claims": "There is no theoretical claim.", "experimental_designs_or_analyses": "I did check all of them. Only weak part is the claims regarding the synthetic datasets. So perhaps this is attributed to the way that the synthetic datasets are generated. Autoregression and interrelatedness are too simple, more free form function versions (inverse relations), nonlinear relations, sinusoidal seasonalities, or aperiodic cycles could justify the results better.", "relation_to_broader_scientific_literature": "It is related to  causal discovery using transformer architectures e.g.,  [Kong, Lingbai, et al. \"CausalFormer: An Interpretable Transformer for Temporal Causal Discovery.\" IEEE Transactions on Knowledge and Data Engineering (2024).]  since this very much depends on the transformer architecture taken into account.", "essential_references_not_discussed": "Temporal Fusion Transformer (Lim, Bryan, et al. \"Temporal fusion transformers for interpretable multi-horizon time series forecasting.\" International Journal of Forecasting 37.4 (2021): 1748-1764.)", "other_strengths_and_weaknesses": "Paper is very well written,  so the clarity is good. The work is original in the sense of methodology,  and quite significant, since there was quite many debates on the usefulness of transformers in forecasting with more traditional forecasting side was more skeptical. It can  definitely be influential in understanding transformers for forecasting. \n\n The paper also sheds light on the influential controversial paper showing some linear functions do better in forecasting (Zeng, A., Chen, M., Zhang, L., and Xu, Q. Are transform- ers effective for time series forecasting? In Proceed- ings of the AAAI conference on artificial intelligence, volume 37, pp. 11121–11128, 2023.) \n\nThe role of interdependencies between variates per data set, on the role of performance of certain architecture, is a message that makes utter sense. \n\nMoreover, although kept a bit short, the finding of Avg Var Corr not being very crucial metric, and the effect of Z-normalisation  in TIHM and MINDEr datasets are important findings.", "other_comments_or_suggestions": "Include TFT in the analysis. Try to come up with a better synthetic data generation that can pronounce your claims stronger. (if they are really the case.)\n\nSince mutual information metric is crucial it needs better explanation.", "questions_for_authors": "1)Why did you not include TFT (perhaps the first model) in your results? (it is a highly-cited and commonly practiced successful model.)\n2)What is the major role of MAX Mi?  Does it give us some extra information? \n3) Could you explain \\sigma ij, at the lowest level,  is it simply correlation?  (the N sample thing does not fully click in my mind, and the way they are selected.)", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sShEvqHCNn", "reviewer": "Reviewer_jMHQ", "summary": "The paper focuses on a recently heated topic and an important topic in time series forecasting -- which is to conduct further evaluation on the previously proposed models, not only on how a model can improve the forecasting performance, but also consider the performance changes are potentially related to the characteristics of the datasets. While the paper does not introduce a novel model architecture, it provides valuable insights into the generalization ability of Transformer-based models beyond widely used benchmark datasets (lots of them are poorly designed, see **Other Strengths And Weaknesses**).", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your thoughtful comments. We truly appreciate your effort and time for reviewing our work.\n\n*Response to comments regarding additional metrics for evaluating forecasting accuracy:*\n\n&nbsp;&nbsp;&nbsp;&nbsp;    In this work, our primary focus is on understanding how different token representations within the attention mechanism lead to significant variations in model performance. We report MAE and MSE because they are the most widely used evaluation criterion in the literature, allowing for easy comparison with the related works. We acknowledge that this is not sufficient to evaluate models comprehensively. This is also the reason we introduce several mutual information-based scores in this work to explore model behaviour in capturing interactions between variates. The broader question of how to design and apply appropriate evaluation metrics for time series forecasting is a substantial topic in itself, requiring holistic analysis of both datasets and model behaviour. We consider this a valuable direction for future research.\n\n*Response to comments regarding the inclusion of additional datasets:*\n\n&nbsp;&nbsp;&nbsp;&nbsp;    We fully agree that the benchmark datasets commonly used for evaluating time series forecasting models should be significantly expanded. This is one of the key points we aim to highlight through our work. In this study, we selected the most widely used datasets in the literature, as they have become a standard reference for comparing model performance. However, based on our analysis, these benchmarks are limited in scope and do not sufficiently reflect the diversity of real-world forecasting challenges. To address this, we present results not only on these standard datasets but also on eight synthetic datasets and two real-world healthcare datasets, which help reveal the limitations of relying solely on several homogeneous benchmarks. We appreciate the suggestion and will examine the recommended datasets in our experimental settings. \n\n*Response to comments regarding essential references:*\n    \n&nbsp;&nbsp;&nbsp;&nbsp;    Thank you for the valuable suggestion. We agree that considering dataset characteristics and appropriate modelling strategies—like traditional methods such as ARIMA—can greatly benefit modern time series forecasting. We see this as an important direction for future work, which should ideally be addressed alongside the analysis of dataset characteristics and the design of suitable evaluation metrics. We'll acknowledge this point in the revised manuscript.\n\n*Response to questions:*\n\n1. Limited Complexity -- Why only two variates included in the synthetic data?\n\n    We used only two variates in the synthetic data to maintain clarity in analyzing inter-variate dependencies. We had considered to generate more variates, however, it made the analysis of results much less clear. For instance, introducing a third variate would require defining its dependency on the other two, resulting in two additional hyperparameters. Moreover, since the second variate is already dependent on the first, the dependency of the third variate would effectively be influenced by three parameters. This added complexity would make the analysis less transparent and interpretable.\n\n2. Simplified Dependency Structure -- Why only linear dependencies between variates in the synthetic data?\n\n    Similarly, linear dependencies are straightforward to interpret and analyze. We had considered increasing the complexity of the dependencies if the models demonstrated strong performance in capturing linear relationships. However, even with these simple linear dependencies, we observed that patch-wise and variate-wise models still rely heavily on intra-variate dependency except in cases where the inter-variate dependency is extremely high (e.g. $\\alpha = 0.8$). Given this, we did not see a clear benefit in introducing more complex dependencies at this stage.", "claims_and_evidence": "See limitation with the **experimental designs** and **questions** to better support the claims.", "methods_and_evaluation_criteria": "The evaluation is conducted by averaging across the forecasting horizon, but a more comprehensive approach would be to assess also the model’s predictability for each forecasting horizon independently, as recommended in time series literature [1,2]. This is especially important for a paper claiming to reduce the error accumulation over the forecasting horizon.\n\n[1] Another look at measures of forecast accuracy.\n[2] Forecasting: Principles and Practice. Chapter 3.", "theoretical_claims": "See limitation with the **experimental designs** and **questions** to better support the claims.", "experimental_designs_or_analyses": "The experiments primarily rely on benchmark datasets that have already been used to evaluate the selected Transformer models. From this perspective, this paper is mainly a reproduction of the previous models on the already-tested data sets. However, these benchmark datasets alone are insufficient to draw a comprehensive understanding of how Transformers perform in time series forecasting (see Weaknesses). It would be more valuable to include additional high-quality datasets, particularly from diverse domains such as weather forecasting (weather bench etc.), power price prediction (see power price [1] used in TimeXer[2]), and other datasets that have long been established in specialized fields. This would help assess the models’ robustness and generalization across different types of time series data.\n\n- [1] Forecasting day-ahead electricity prices: A review of state-of-the-art algorithms, best practices and an open-access benchmark\n- [2] TimeXer: Empowering Transformers for Time Series Forecasting with Exogenous Variables", "relation_to_broader_scientific_literature": "see *Essential References*", "essential_references_not_discussed": "There has been work discussing how data set characteristics could influence the model's predictive capacity [1]. The traditional methods have also included a lot of modelling / preprocessing designs for different data sets (eg. ARIMA with differencing for non-stationary data sets). Such modelling design should also be considered with modern time series forecasting in transformer models, and it would be of greater impact of this paper to provide a more comprehensive guidance towards how data characteristics can guide the development / choice of deep learning models.\n\n- [1]: Forecast Evaluation for Data Scientists: Common Pitfalls and Best  Practices", "other_strengths_and_weaknesses": "**Data set:**\nIt has been argued that current benchmark datasets lack general predictive power (see: https://cbergmeir.com/talks/neurips2024/). For instance, the weather dataset requires a longer training time span (like PanguWeather[1], Aurora[2], ClimaX[3] etc.) and more external indicators and that the electricity load in the ETT dataset is also closely related to the weather, which potentially could benefit from the inclusion of weather indicators. The reliance on such low-quality datasets raises concerns about the generalisability of the conclusions drawn in this paper to real-life scenarios. A good thing about this paper, is that they also include a synthetic data set, which to some extent alleviates such concern.\n\n- [1] Pangu-Weather: A 3D High-Resolution Model for Fast and Accurate Global Weather Forecast\n- [2] Aurora: A Foundation Model of the Atmosphere\n- [3] ClimaX: A foundation model for weather and climate", "other_comments_or_suggestions": "see **questions for authors**", "questions_for_authors": "**Synthetic data**\nThe use of synthetic datasets is a promising approach to explore how dataset characteristics affect model performance. However, several aspects of the dataset design would benefit from further clarification:\n- **Limited Complexity**: The current design includes only two variates, which may not adequately reflect the complexity of real-world multivariate time series that often involve higher dimensionality and more intricate interactions. Could the authors clarify on the rationale behind this choice and whether extending the dataset to more variates was considered?\n- **Simplified Dependency Structure**: The dependency parameter $\\alpha$ appears to introduce only linear dependencies between the variates. This design may not sufficiently represent the diversity of relationships observed in real-world data, where dependencies are often non-linear, lagged, or state-dependent. Could the authors clarify whether non-linear or dynamic dependencies were considered, and how the dataset design could be extended to better capture such patterns?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mC5WZ1KpRs", "reviewer": "Reviewer_U2oa", "summary": "The paper investigates the effectiveness of Transformer-based models for time series forecasting, focusing on why simpler Transformers outperform more complex ones. These findings include that intra-variate dependencies dominate the performance of existing forecasting benchmarks, tokenization/channel independence are critical techniques to capture intra-variate patterns, and Z-score normalization significantly improves performance on non-stationary forecasting benchmarks. The authors delve into several representative Transformers on commonly adopted benchmarks and synthetic datasets.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your comments. We truly appreciate your effort and time for reviewing our work.\n\n*Response to the comment challenging the assumption that the model's output is deterministic:*\n\n&nbsp;&nbsp;&nbsp;&nbsp;    While MSE is linked to the assumption of Gaussian noise during training, it does not make the model's predictions stochastic. In this case, the model trained with MSE produces a single, fixed output for a given input. To obtain truly probabilistic or stochastic outputs, one would need to model and sample from the predictive distribution explicitly, such as through Bayesian methods or by estimating parameters of a distribution and sampling accordingly.\n\n*Response to the comment regarding pre-trained Transformers such as Time-MoE and Chronos:*\n\n&nbsp;&nbsp;&nbsp;&nbsp;    As stated at the end of Sec. Related Work, our study focuses on lightweight transformer architectures that are designed to be trained from scratch for an individual dataset. We did not include pre-trained models as they typically follow different training protocols and involve more complex architectures, making fair and transparent comparisons difficult. Additionally, Time-MoE uses a decoder-only architecture, which is fundamentally different from the encoder-based structures used in the lightweight models. Chronos, on the other hand, tokenizes time series into discrete bins through simple scaling and quantization, a strategy that differs significantly from the point-wise, patch-wise, and variate-wise tokenization. We appreciate the suggestion and will include references in the revision.\n\n*Response to the comment regarding broader validation of the importance of skip-connection:* \n\n&nbsp;&nbsp;&nbsp;&nbsp;    We conducted the skip-connection ablation study to explore the following questions (Line 327): “Why do transformers with basic attention mechanisms perform well in time series forecasting? Which components of the basic transformer architecture contribute most to this success?” Our focus is specifically on **basic and effective** transformer architectures. iTransformer meets both criteria, and we consider its structure representative for understanding key design components in transformer-based models. We analyzed skip connections in iTransformer because we hypothesized that they play a key role in learning intra-variate patterns, especially within an architecture designed to support inter-variate attention. Moreover, in models using point-wise tokens, skip connections primarily enhance inter-variate patterns, which differs fundamentally from the goal of this ablation study.\n\n\n*Response to comments regarding additional references.*\n\n&nbsp;&nbsp;&nbsp;&nbsp;    Thank you for your suggestions. While we agree that the suggested papers are relevant to the broader topic, we believe that the specific findings of our work are **not** addressed in them. \n\n&nbsp;&nbsp;&nbsp;&nbsp;    [1] Rethinking Channel Dependence... ICLR 2024\n\n&nbsp;&nbsp;&nbsp;&nbsp;    [1] proposes LIFT, a plugin method designed to identify and utilize locally stationary lead-lag relationships between variates to improve forecasting performance. While it focuses on modeling lead-lag dependencies, our work evaluated the effectiveness of various models in capturing general inter-variate relationships. The mutual information-based scores we propose are model-agnostic and capable of assessing inter-variate dependencies without being limited to specific relationship types. \n\n&nbsp;&nbsp;&nbsp;&nbsp;    [2] Reversible Instance Normalization... ICLR 2022.\n\n&nbsp;&nbsp;&nbsp;&nbsp;    Reversible Instance Normalization (RevIN), a method similar to Z-score normalization, was designed to address distributional shifts in time series data. A key difference is that RevIN incorporates learnable parameters within its normalization process, whereas the Z-score normalization examined in our study is the standard, non-learnable version. Moreover, our findings **contradict** the suggestion that \"Z-Norm addresses distributional shift.\" Specifically, we observed that Z-score normalization degraded model performance on synthetic datasets which are non-stationary with monotonic trends (Table 4).\n\n*Response to other suggestions and questions:*\n\n&nbsp;&nbsp;&nbsp;&nbsp;    1. \"...include error bars\", \"...without multiple runs\", \"...provide training configurations\".\n\n&nbsp;&nbsp;&nbsp;&nbsp;    As stated in Sec. Experiments (Line 216 - 219): \"All the experimental results in this work are averaged over 3 runs with different random seeds. Standard deviation of the results are provided in the Appendix.\" Both Fig. 3 and 4 include error bars. We will add hyperprameter configurations in the appendix in the revised manuscript.\n\n&nbsp;&nbsp;&nbsp;&nbsp;    2. \"...provide showcases of synthetic datasets\".\n\n&nbsp;&nbsp;&nbsp;&nbsp;    We will add visualization of synthetic datasets in the revised manuscript.", "claims_and_evidence": "> The assumption $H\\left(\\hat{\\mathbf{x}}_j \\mid \\mathbf{x}_i, \\mathbf{x}_{/ i}\\right)=0$ (line 166)\n\nThis assumption is not right. The authors regard the model as a deterministic forecaster. However, the model is trained with probabilistic distributions, e.g., MSE specifies a Gaussian distribution.\n\n> Point-wise Transformers underperform due to their weak capability of capturing intra-variate patterns (supported by Intra MI scores and synthetic experiments).\n\nThe claim that Transformers are less effective at capturing patterns within a single variate may not only stem from their tokenization (counterexamples such as recent pre-trained Transformers Time-MoE and Chronos). In this regard, the authors should also consider the sufficiency of training samples.\n\n> Skip connections are crucial for capturing intra-variate dependencies but unsuitable to capture inter-variate patterns. \n\nThis conclusion can be highly dependent on the selected architecture of iTransformer. Broader validation on more types of Transformers (e.g., Transformers in Table 1) is needed.", "methods_and_evaluation_criteria": "* The MI-based metrics assume deterministic outputs, which may not hold for probabilistic distribution for optimization.", "theoretical_claims": "See above.", "experimental_designs_or_analyses": "See Claims And Evidence.", "relation_to_broader_scientific_literature": "Several of the authors' findings in this paper have been mentioned in previous works: \n\n* About the influence of Channel Independence: Rethinking Channel Dependence for Multivariate Time Series Forecasting: Learning from Leading Indicators. ICLR 2024.\n* Z-Norm addresses distributional shift: Reversible Instance Normalization for Accurate Time-Series Forecasting against Distribution Shift. ICLR 2022.", "essential_references_not_discussed": "The aforementioned works should also be cited.", "other_strengths_and_weaknesses": "* Strength: The metric of mutual information for time series forecasting is original.\n* Weakness: (1) The findings presented by this paper need more comprehensive validation (e.g., evaluation on short-term forecasting datasets) (2) The paper can be further improved by providing insights into the solution to address these issue. (3) The description of data synthesis takes up too much space in the main text, while the conclusions drawn from the experiments are presented in a disorganized manner.", "other_comments_or_suggestions": "Suggestion: (1) Include error bars in evaluations to assess variance. (2) Instead of assessing different types of Transformers, presenting respective ablation studies on disentangled components (e.g., tokenization, attention mechanism, channel independence, and Z-norm) can exclude their mutual influence. (3) It can be more helpful to provide showcases of synthetic datasets (under different configurations $\\alpha$ $\\gamma$) to illustrate the pipeline of Figure 1.", "questions_for_authors": "The main conclusions are drawn by comparing the model's performance without multiple runs. Is the performance mutable and easily influenced by the training process? The authors need to provide the training configurations and error bars (at least in the appendix).", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper studies the performance of various transformer architectures in time series forecasting, considering the impact of point-wise, patch-size, and 'variate'-wise tokenization, intra and inter-variate attention, and other components. 

Results suggest that most time series benchmarks predominantly require modeling intra-variate patterns (rather than inter-variate patterns), and that normalization and skip-connections are crucial. These results are further supported by experiments on controlled synthetic datasets.

The paper received supportive scores from 3 out of 4 reviewers. Reviewers noted that the experiments were well designed, the paper was clear and it shed light on some surprising previous findings, e.g. 'Are transformers effective for time series forecasting? at AAAI. 
Some weaknesses were identified that can be easily fixed, such as adding missing citations, and clarifying the limitations of the study (e.g., restricted to transformers trained on a single dataset, rather than so-called time series foundation models).

I agree with the majority of reviewers that the paper should be accepted. I encourage the authors to consider all feedback in their final camera-ready submission.
Decision: Accept (poster)