iTransformer: Inverted Transformers Are Effective for Time Series Forecasting
Authors: Yong Liu
Tengge Hu
Haoran Zhang
Haixu Wu
Shiyu Wang
Lintao Ma
Mingsheng Long
Venue: ICLR 2024 spotlight (2024)
Keywords: Time Series Forecasting
Transformer
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: Based on the reflection on the duties of Transformer components, we propose inverted Transformer for time series forecasting, which achieves the SOTA in real-world applications and shows powerful strength on framework generalization.

Abstract: The recent boom of linear forecasting models questions the ongoing passion for architectural modifications of Transformer-based forecasters. These forecasters leverage Transformers to model the global dependencies over temporal tokens of time series, with each token formed by multiple variates of the same timestamp. However, Transformers are challenged in forecasting series with larger lookback windows due to performance degradation and computation explosion. Besides, the embedding for each temporal token fuses multiple variates that represent potential delayed events and distinct physical measurements, which may fail in learning variate-centric representations and result in meaningless attention maps. In this work, we reflect on the competent duties of Transformer components and repurpose the Transformer architecture without any modification to the basic components. We propose iTransformer that simply applies the attention and feed-forward network on the inverted dimensions. Specifically, the time points of individual series are embedded into variate tokens which are utilized by the attention mechanism to capture multivariate correlations; meanwhile, the feed-forward network is applied for each variate token to learn nonlinear representations. The iTransformer model achieves state-of-the-art on challenging real-world datasets, which further empowers the Transformer family with promoted performance, generalization ability across different variates, and better utilization of arbitrary lookback windows, making it a nice alternative as the fundamental backbone of time series forecasting. Code is available at this repository: https://github.com/thuml/iTransformer.

Published as a conference paper at ICLR 2024
ITRANSFORMER: INVERTED TRANSFORMERS ARE EFFECTIVE FOR TIME SERIES FORECASTING
Yong Liu," Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang Lintao Ma², Mingsheng Long School Software, BNRist, Tsinghua University, Beijing 100084, China Ant Group, Hangzhou, China
{liuyong21, htg21, 2-hr20, whx20 fma ils. www.ttsiiinnccc..e cdu. tweimings .wy,, lintao... mlt mentteentrouuu com, mingsheng@ts edu.
ABSTRACT
The recent boom linear forecasting models questions the ongoing passion for architectural modifications of Transformer-basee forecasters. These forecasters leverage Transformers to model the global dependencies over temporal tokens time series, with each token formed by multiple variates of the same timestamp. However, Transformers are challenged in forecasting series with larger lookback windows due performance degradation and computation explosion. Besides, the embedding for each temporal token fuses multiple variates that represent potential delayed events and distinct physical measurements, which may fail in learning variate-centric representations and result in meaningless attention maps. In this work, we reflect on the competent duties of Transformer components and repurpose the Transformer architecture without any modification to the basic components. We propose iTransformer that simply applies the attention and feed-forward network the inverted dimensions. Specifically. the time points of individual series em- bedded into variate tokens which are utilized by the attention mechanism to capture multivariate correlations; meanwhile, the feed-forward network is applied for each variate token to learn nonlinear representations. The iTransformer model achieves state-of- state-of-the-art on challenging real-world datasets, which further empowers the Transformer family with promoted performance, generalization ability across differ- ent variates, and better utilization of arbitrary lookback windows, making a nice alternative as the fundamental backbone of time series forecasting. Code is avail- wwwpaammmmrttaaaffraomoeo
able this repository: https: //github thub.
INTRODUCTION
Transformer Vaswani et al. 2017) has achieved tremendous suc- cess natural language processing (Brown et al.) 2020) and computer vision (Dosovitskiy et al. 2021), growing into the foun- dation model that follows the scaling law (Kaplan et al.) 2020). Inspired by the immense success in extensive fields, Transformer with strong capabilities of depicting pairwise dependencies and extracting multi-level representations in sequences is emerging in time series forecasting [Wu et al.) 2021 Nie et al. 2023).
ETT
PEMS
0,33
Soler
pleh
However, researchers have recently begun question the validity of Transformer-based forecasters, which typically embed multiple variates of the same timestamp into indistinguishable channels and apply attention on these temporal tokens to capture temporal dependencies. Considering the numerical but less semantic rela- tionship among time points, researchers find that simple linear layers, which can be traced back to statistical forecasters (Box & Jenkins 1968), have exceeded complicated Transformers on both ct al.] 2023 Das et al. 2023). Meanwhile, ensuring the independence of variate and utilizing mutual
Trailic
Weather
[Brniormer
Dlins
Figure Performance of iTrans- former. Average results (MSE) are reported following TimesNet (2023 performance and efficiency ZZng
"Equal Contribution

---

Published as a conference paper at ICLR 2024
Timee Stege
Value
Token
Token
Token
Transformer View
Emhodd
Mised Represmnttion
Token
Temporal
over
Representations
Ercheddingg
Token
Vallue
Tokenn
iransformer View
Variates
Voriate Liumived Reprosmnttion
Token
Time
Attention Variate Tokres
Series Reprosentatioss
Figure 2: Comparison between the vanilla Transformer (top) and the proposed iTTranffrmeer (bottom). Transformer embeds the temporal token, which contains the multivariate representation of each time step. i'Transfommer embeds each series independently the variate token, such that the attention mod- ule depicts the multivariate correlations and the feed-forward network encodes series representations.
information is ever more highlighted by recent research that explicitly models multivariate correla- tions to achieve accurate forecasting Zhang & Yan 2023 Ekambaram et al. 2023), but this goal can be hardly achieved without subverting the vanilla Transformer architecture.
Considering the disputes of Transformer-basse forecasters, reflect why Transformers perform even worse than linear models in time series forecasting while acting predominantly in many other forecasters may be not suitable for
fields. We notice that the existing structure of Transformer-bant multivariate time series forecasting. As shown on the top of Figure 2 it is notable that the points of the same time step that basically represent completely different physical meanings recorded by inconsistent measurements are embedded into one token with wiped-out multivariate correlations. And the token formed by a single time step can struggle reveal beneficial information due to excessively local receptive field and time-unaligned events represented by simultaneous time points. Besides, while series variations can be greatly influenced by the sequence order, permutation- invariant attention mechanisms are improperly adopted on the temporal dimension (Zeng al.) 2023). Consequently, Transformer is weakened to capture essential series representations and portray mu multivariate correlations, limiting its capacity and generalization ability on diverse time series data.
Concerning the potential risks embedding multivariate points of a timestamp as (temporal) token, we take an inverted view time series and embed the whole time series of each variate independently into a (variate) token, the extreme case of Patching (Nie et al. 2023) that enlarges local receptive field. By inverting, the embedded token aggregates the global representations of series that can be more variate-centric and better leveraged by booming attention mechanisms for multivariate correlating. Meanwhile, the feed-forward network be proficient enough learn generalizable representations for distinct variates encoded from arbitrary lookback series and decoded to predict future series.
Based on the above motivations, we believe is not that Transformer is ineffective for time series forecasting, but rather it is improperly used. In this paper, we revisit the structure of Transformer and advocate (Transformer as a fundamental backbone for time series forecasting. Technically, we embed each time series as variate tokens, adopt the attention for multivariate correlations, and employ the feed-forward network for series representations. Experimentally, the proposed iTransformer achieves state-of-the-ate performance on real-world forecasting benchmarks shown in Figure and surprisingly tackles the pain points of Transformer- based forecasters. Our contributions lie in three aspects:
We reflect on the architecture of Transformer and refine that the competent capability of native Transformer components on multivariate time series is underexplored.
We propose iTransformer that regards independent time series as tokens capture multivari- ate correlations by self-attention and utilize layer normalization and feed-forward network modules to learn better series-global representations for time series forecasting.
Experimentally, iTransformer achieves comprehensive state-of-the- tof-the-art real-world bench- marks. We extensively analyze the inverted modules and architecture choices, indicating a promising direction for the future improvement of Transformer-based forecasters.
2

---

Published as a conference paper at ICLR 2024
2
RELATED WORK
With the progressive breakthrough made in natural language processing and computer vision areas, elaboratively designed Transformer variants are proposed tackle ubiquitous time series forecasting applications. Going beyond contemporaneous TCNs (Bai et al. 2018: Liu et al. (022a) and RNN- based forecasters (Zhao al. 2017 Rangapuram al. 2018 Salinas al. (020). Transformer has exhibited powerful sequence modeling capability and promising model scalability, leading to the trend of passionate modifications adapted for time series forecasting.
Through a systematical review of Transformer-basse based forecasters, we conclude that existing modifi- cations can be divided into four categories by whether to modify the component and architecture. As shown in Figure 3] the first category (Wu et al.) 2021 Li et al. 2021 Zhou et al. [022). which the most common practice, mainly concerns the component adaptation, especially the attention module for the temporal dependency modeling and the complexity optimization on long sequences. Nevertheless, with the rapid emergence of linear forecasters (Oreshkin al. 2019 Zeng et al.,
2023 Das et al. 2023 Liu et al. 2023), the impressive performance and efficiency continuously challenge this direction. Soon afterward, the second category attempts to fully utilize Transformer. It pays more attention to the inherent processing of time series, such Stationarization (Liu et al.) 20226). Channel Independence, and Patching (Nie et al., 2023), which bring about consistently improved performance. Moreover, faced with the increasing significance of the independence and mutual interactions of multiple variates, the third category refurbishes Transformer in both aspects of component and architecture. Representative (Zang & Yan 2023) explicitly captures the cross-time and pross-variate dependencies by the renovated attention mechanism and architecture.
Unlike previous works, iTransformer modifies none of the native components of Transformer. Instead, we adopt the components on the inverted dimensions with the altered architecture, as the only one that belongs the fourth category to our best knowledge. We believe the capabilities of the components have stood the test extensively, the truth is that the architecture of Transformer is improperly adopted.
(l) Autoformer, Informer....
Modified Component
(II Crossformer....
Feed-forward
Modified Attn
No Modified Architecture
Modified Architecture
Series Processing
Transformer
Original Arch
No Modified
Mudifiedd Arch
(II) Patch TST, NSTransformer...
Component
(IV) iTransformer (Ours)
Figure 3: Transformer-based forecasters categorized by component and architecture modifications.
ITRANSFORMER
In multivariate time series forecasting, given historical observations X {x1, (xi,...,xy) RTNN with T time steps and N variates, we predict the future S time steps = {xT=-1..- XT+S) E RSNN, For convenience, we denote X,; as the simultaneously recorded time points the step 4, and X.. as the whole time series of each variate indexed by n. It is notable that Xt,; may not contain time points that essentially reflect the same event in real-world scenarios because of the systematical time lags among variates in the dataset. Besides, the elements of X,,: can be distinct from each other physical measurements and statistical distributions, for which variate X generally shares.
3.1 STRUCTURE OVERVIEW
Our proposed iTransfomme illustrated in Figure 4 adopts the encoder-only architecture of Trans- former (Vaswani et al. 2017), including the embedding, projection, and Transformer blocks.
Embedding the whole series the token Most Transformer- based forecasters typically regard multiple variates of the same time as the (temporal) token and follow the generative formulation of forecasting tasks. However, we find the approach the numerical modality can be less instructive for
3

---

Published as a conference paper at ICLR 2024
(d)
Outpat
Features
Projection
TepBlork
LayerNorm
1 Temporal LayerNorm
Variate
Feed-forward
Dense
Drop
x
LayerNorm
(b)
Query
Multivariate
Multivariate
wwww
Jariate
MatMul
Scale
Correlations
Map
MMY
Key
Embedding
Embredded
Embedding
Value
MatMul
Series
Input
Figure 4: Overall structure of iTransformer, which shares the same modular arrangement with the encoder of Transformer. (a) Raw series of different variates are independently embedded as tokens. (b) Self-attention is applied to embedded variate tokens with enhanced interpretability revealing multivariate correlations. (c) Series representations each token are extracted by the shared feed- forward network. (d) Layer normalization is adopted reduce the discrepancies among variates.
learning attention maps, which is supported by increasing applications of Patching (Dosovitskiy et al. 2021 Nie et al. 2023) that broadens the respective field. Meanwhile, the triumph of linear forecasters also challenges the necessity of adopting heavy encoder-decoder Transformer for generating tokens. Instead, our proposed encoder-only iTransformer focuses on representation learning and adaptive correlating of multivariate series. Each time series driven by the underlying complicated process is firstly tokenized describe the properties of the variate, applied by self-attention for mutual interactions, and individually processed by feed-forward networks for series representations. Notably, the task to generate the predicted series is essentially delivered to linear layers, which has been proven competent by previous work (Das et al. 2023) and we provide detailed analysis in the next section.
Based on the above considerations, in iTransformer, the process of predicting future series of each specific variate Y in based on the lookback series X is simply formulated follows:
h Embedding (..)))
H ++ TrmBlock (H'), 1= 0, O......4
Y Projection(h.)
where H (hi..... hy) RND contains N embedded tokens of dimension D and the su- perscript denotes the layer index. Embedding : RT + TRD and Projection : RDD + IR' are both implemented by multi-layer perceptron (MLP). The obtained variate tokens interact with each other by self-attention and are independently processed by the shared feed-forward network each TrmBlock. Specifically, the order of sequence implicitly stored in the neuron permutation the feed-forward network, the position embedding in the vanilla Transformer is no longer needed here.
iTransformers The architecture essentially presupposes no more specific requirements on Trans- former variants, other than the attention is applicable for multivariate correlation. Thus, bundle of efficient attention mechanisms (Li et al. 2021 Wu et al. 2022 Dao et al. 2022) can the plugins, reducing the complexity when the variate number grows large. Besides, with the input flexibility of attention, the token number can vary from training to inference, and the model is allowed to be trained on arbitrary numbers of variates. The inverted Transformers, named iTransformers, are extensively evaluated in experiments of Section 4.2 and demonstrate advantages on time series forecasting.
3.2 INVERTED TRANSFORMER COMPONENTS
We organize a stack of L blocks composed of the layer normalization, feed-fore forward network, and self-attention modules. But their duties on the inverted dimension carefully reconsidered.
4

---

Published as a conference paper at ICLR 2024
Layer normalization Layer normalization (Ba et al. the convergence and training stability of deep networks. In typical Transformer based forecasters, the module normalizes the multivariate representation of the same timestamp, gradually fusing the variates with each other. Once the collected time points do not represent the same event, the operation will also introduce interaction noises between noncausal or delayed processes. In our inverted version, the normalization is applied to the series representation of individual variate as Equation 2 which has been studied and proved effective in tackling non-stationary problems (Kim al., 202 Liu et al. 2022b). Besides, since all series as (variate) tokens are normalized to a Gaussian distribution, the discrepancies caused by inconsistent measurements can be diminished. By contrast, in previous architecture, different tokens of time steps will be normalized, leading to oversmooth time series.
2016 is originally proposed to increase
LayerNorm(H) ={
hn Mean(h..) Var(hh)
n= 1= N
Feed-forward network Transformer adopts the feed-forward network (FFN) as the basic building block for encoding token representation and is identically applied to each token, As aforementioned. in the vanilla Transformer, multiple variates of the same timestamp that form the token can be malpositioned and too localized to reveal enough information for predictions. In the inverted version, FFN is leveraged on the series representation of each variate token. By the universal approximation theorem (Horiik) 1991), they can extract complicated representations to describe a time series. With the stacking of inverted blocks, they are devoted to encoding the observed time series and decoding the representations for future series using dense non-linear connections, which work effectively as the recent works completely built on MLPs (Tolstikhin et al. 2021 Das et al. 2023).
More interestingly, the identical linear operation on independent time series, which serves as the combination of the recent linear forecasters (Zeng et al. 2023) and Channel Independence (Nic et al. 2023), can be instructive for us to understand the series representations. Recent revisiting on linear forecasters (Li et al.) 2023) highlights that temporal features extracted by MLPs supposed to be shared within distinct time series. We propose a rational explanation that the neurons of MLP are taught to portray the intrinsic properties of any time series, such as the amplitude, periodicity, and even frequency spectrums (neuron as filter), serving as more advantageous predictive representation learner than the self-attention applied on time points, Experimentally, we validate that the division of labor helps enjoy the benefits of linear layers in Section 4.3) such as the promoted performance if providing enlarged lookback series, and the generalization ability on unseen variates.
Self-attention While the attention mechanism is generally adopted for facilitating the temporal dependencies modeling in previous forecasters, the inverted model regards the whole series of one variate as an independent process. Concretely, with comprehensively extracted representations of each time series H (ho,... hr) ) E RR-D the self-attention module adopts linear projections to get queries, keys, and values Q, V E Nxde where dę is the projected dimension.
With denotation of q: k, RRde as the specific query and key of one (variate) token, we notice that each entry of the pre-Softmax scores is formulated as Ai,j = (QKT /Vdr).s x q. ky. Since each token is previously normalized on its feature dimension, the entries can somewhat reveal the variate-wise correlation, and the whole score map A N exhibits the multivariate correlations between paired variate tokens. Consequently, highly correlated variate will be more weighted for the next representation interaction with values V. Based on this intuition, the proposed mechanism is believed to be more natural and interpretable for multivariate series forecasting. We further provide the visualization analysis of the score map in Section 4.3 and Appendix E.T
EXPERIMENTS
We thoroughly evaluate the proposed iTTransformer on various time series forecasting applications, validate the generality of the proposed framework and further dive into the effectiveness of applying the Transformer components on the inverted dimensions of time series.
Datasets We extensively include real-world datasets in our experiments, including ECL, ETT (4 subsets), Exchange, Traffic, Weather used by Autoformer (Wu et al. 2002), Solar-Fnergy datasets
5

---

Published as a conference paper at ICLR 2024
proposed in LSTNet (Lai al. 2018), and PEMS (4 subsets) evaluated in SCINet (Liu et al. 2022a).
We also provide the experiments on Market (6 subsets) in Appendix F.4 It records the minute- sampled server load of Alipay online transaction application with hundreds of variates, where consistently outperform other baselines. Detailed dataset descriptions are provided in Appendix A.
4.1 FORECASTING RESULTS
In this section, we conduct extensive experiments to evaluate the forecasting performance of our proposed model together with advanced deep forecasters.
Baselines We carefully choose 10 well-acknowledged forecasting models as our benchmark,
including (1) Transformer-basedd methods: Autoformer (Wu et al. (021), FEDformer (Zhou et al.
2022), Stationary (Liu et al. 20226), Crossformer (Zhang & Yan 2023), PatchTST (Nie et al. 20233
(2) Linear-baut based methods: DLinear (Zeng et al., 2023). TiDE (Das et al. 2023). RLinear (Li et al.
2023 ; and (3) TCN-based methods: SCINet (Liu et al. 2022a). TimesNet (Wu et al., 2023
Main results Comprehensive forecasting results are listed in T with the best red and the second underlined. The lower MSE/MAE indicates the more accurate prediction result. Compared with other forecasters, iTransformer is particularly good at forecasting figh-dimensiona time series. Besides, PStchTST the previous state-of-the-art fails in many cases of PEMS, which can stem from the extremely fluctuating series of the dataset, and the patching mechanism of PatchTST may lose focus on specific locality to handle rapid fluctuation. By contrast, the proposed model aggregating the whole series variations for series representations can better cope with this situation. Notably, as the representative that explicitly captures multivariate correlations, the performance of Crossformer is still subpar to i'ransformer, indicating the interaction of time-unaligned patches from different multivariate will bring about unnecessary noise for forecasting. Therefore, the native Transformer components are competent for temporal modeling and multivariate correlating, and the proposed inverted architecture can effectively tackle real-world time series forecasting scenarios.
Table 1: Multivariate forecasting results with prediction lengths S E {12, 24, 36, 48) for PEMS and S E (96, 192, 336, 720} for others and fixed lookback length = 96. Results are averaged from all prediction lengths. Avg means further averaged by subsets. Full results are listed in Appendix E44
Models
iTranformer
RI innnn
PacATST
Crossformer
TDD
TimesNer
Dlare
SCINer
FRDiormer
Stationary
AntnFermr
(Ours)
2023
2023
2025
20233
2023
2022
20220
Metric
MSE
MAE MSE
MAE
MSE
MAE]
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAL
MSE
MAH
MSE MAE MSE MAE
ECL
0,178 0.270
(02199
0.295|0.205
0.200
0.334 0.334
0.344 0.344]|..12 0.29
(0.212
0.200 (0.268 0.365 (0.214 0.327 (0.193 0,296 0.227 0,338
ETT (Avg) 0.383 0.399 |0.380 0.392/0.181
6.397
6..483 (8.482 o 4770/0.39) 0.4040.4442
0.444 (o. 689 o 397 [0 0.428 (0.47) 0.464 0.465 0.459
Exchange 0,350
0.403
jo.388 0.41770.367 0.004 o. 20.300 0.413 0.443 (0.354 0.414 75.70 0.626 [0.539 0,429 (0.46] 0.454 (0.133 0,539
Traffic 0.428 0.282
joxk26
0.378/0.481
n. 304 0.500 o. 304 p.304/0.760 760 0.473|0.620 336 (0.625 383 804 in 509 (0.610 0.376 (n (0.624 0.340 (0.638 0.379
Weather 0.278
Jo.222
|.291|0.25
0.281 25.
j0.211
0.287 (0:265 0.317 (0.292 0.363 (0.509 0.360 (0.258 0.314 (03388 0.382
Solar-Energy 0,233 0,262 10.369 0,356|02200
0.307|0.641
0.639 (0.347 (.03700.30 0.319 30.330 0.401 J0.282 0,375 (0.29] 0,381 (0.26] 0.381 0.885 0,711
PEMS (Avg) 0.119 0.218 |a.54 0.422/02117 2.30 j.3445
0.440jo.148
0.246|0. (0.320 0.294
121121 0.222 (0.234 0,327 (0.15] 0.249 00.34 0,575
4.2 ITRANSFORMERS GENERALITY
In this section, we evaluate Trransformers by applying our framework to Transformer and its vari- ants, which generally address the quadratic complexity of the self-attention mechanism, including Reformer (Kitaev et al. 2020), Informer (Li et al. 3. Flowformer (Wu et al., (022) and FlashAt- tention (Dao al. 2022). Surprising and promising discoveries are exhibited, indicating the simple inverted perspective can enhance Transformer-basee forecasters with promoted performance with efficiency, generalization on unseen variates, and better utilization of historical observations.
Performance promotion We evaluate Transformers and the corresponding i'Transfomerrs with the reported performance promotions in Table It is notable that the framework consistently improves various Transformers. Overall, it achieves averaged 38.9 % promotion on Transformer, 36.1% % on Reformer, 28.5% on Informer, 16.8% on Flowformer and 32.2% on Flashformer, revealing the previous improper usage of the Transformer architecture on time series forecasting. Moreover, since the attention mechanism is adopted on the variate dimension in our inverted structure, the introduction of efficient attentions with linear complexity essentially addresses the computational problem due to

---

Published as a conference paper at ICLR 2024
numerous variates, which is prevalent in real- applications but can be resource- resource-consuming consuming for Channel Independence (Nie et al. 2023). Therefore, the idea of iTransformer can be widely practiced on Transformer- based forecasters take advantage of booming efficient attention mechanisms.
Table 2: Performance promotion obtained by our inverted framework. Flashformer means Transformer equipped with hardware-accelerate FlashAttention (Dao et al. 2022). We report the average performance and the relative MSE reduction (Promotion). Full results can be found in Appendix F.2
Transformer
Reformer
Informer (2021)
Flowformer
Flashformer (022)
Models
(2017)
20200)
2022)
Metric
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
Original
0.277
0.372
0.338
0.422
0.311
0.397
0.267
0.359
0.285
0.377
ECL
+Inverted
0.178
0.270
0.208
0,301
0.216
0.311
0.210
0.293
0.206
0.291
Promotion
35.6%
27.4%
38.4%
28.7%
30.5%
21.6%
21.3%
18.6%
27.8%
22.9%
Original
0.665
0.363
0.741
0.422
0.764
0.416
0.750
0,421
0.658
0.356
Traffic
+nverted
0.428
0.282
0.647
0.370
0.662
0.380
0.524
0.355
0.492
0.333
Promotion
35.6%
22.3%
12.7%
12.3%
13.3%
8.6%
30.1% %
15.6%
25.2%
6.4%
Original
0.657
0.572
0.803
0.656
0.634
0.548
0.286
0.308
0.659
0.574
Weather
+Iverted
0.258
0.279
0.248
0.292
0.271
0.330
0.266
0.285
0.262
0.282
Promotion
60.2%
50.8%
69.2%
55.5%
57.3%
39.8%
7.2%
7.7%
60.2%
50.8%
Variate generalization By inverting vanilla Transformers, it notable that the models are empow- ered with the generalization capability on unseen variates. Firstly, benefiting from the flexibility of the number of input tokens, the amount of variate channels is no longer restricted and thus feasible to vary from training and inference. Besides, feed-forward networks are identically applied on independent variate tokens in iTrranffrmmr. As aforementioned, the neurons as filters learn the intrinsic patterns of any time series, which are inclined to be shared and transferable among distinct variates.
To verify the hypothesis, we compare inverting with another generalizing strategy: Channel Indepen- dence, training shared backbone to forecast all variates. We partition the variates of each dataset into five folders, train models with only 20% of variates of one folder, and directly forecast all variates without fine-tuning. We compare the performance in Figure 5 and each bar presents the averaged results of all folders to avoid the randomness of partition. C1-Transformers take a long time predict each variate one by one during inference while iTransformers directly predict all variates and generally present smaller increases, indicating FFN is competent to learn transferable time series representations. leaves potential direction build foundation model upon iTransformer, where diverse multivariate time series with different numbers of variates can be feasibly trained together.
BCL
Traffic
Solar Energy
1.5
+1.12
0.35
15
215
0.5
0.30
1E,00
0.2
1E.44
0.4-
0.30
01
5.06
frstalomr
Flewfomer
Trauisenc
Initexr
Farefernc
Trasafomerr
Infermz
Flowiorme
(20%
(100% valate)
varian))
Figure 5: Performance of generalization unseen variates. We partition the variates of each dataset into five folders, train models with 20% variates, and use the partially trained model to forecast all varieties. iTransformers can be trained efficiently and forecast with good generalizability.
Increasing lookback length Previous works have witnessed the phenomenon that the forecasting performance does not necessarily improve with the increase of lookback length on Transformers (Nie)
al. 2023 Zeng al. 2023), which can be attributed to the distracted attention the growing input. However, the desired performance improvement is generally held on linear forecasts, theoretically supported by statistical methods (Box & Jenkins 1968) with enlarged historical information to be
7

---

Published as a conference paper at ICLR 2024
utilized. As the working dimensions of attention and foed-forward network are inverted, evaluate the performance Transformers and iTransformer in Figure with increased lookback length. The results surprisingly verify the rationality of leveraging MLPs the temporal dimension such that Transformers can benefit from the extended lookback window for more precise predictions.
ECL
Traffic
0.8
0.35
0.7
0.30
MSE
MSE
0.25
0.6
0.20
0.5
0.15
0.4
48
96
192
192
720
Flowformer
Figure 6: Forecasting performance with the lookback length T {48, 96, 192, 336, 720) and fixed prediction length 96. While the performance of Transformer- based forecasters does not necessarily benefit from the increased lookback length, the inverted framework empowers the vanilla Transformer and its variants with improved performance on the enlarged lookback window.
4.3 MODEL ANALYSIS
Ablation study To verify the rational business of Transformer components, we provide detailed ablations covering both replacing components (Replace) and removing components (w/o) experiments. The results are listed in Table iTTrnsformer that utilizes attention on the variate dimension and feed-forward on the temporal dimension generally achieves the best performance. Notably, the performance of vanilla Transformer (the third row) performs the worst among these designs, revealing the potential risks the conventional architecture, which we describe detail in Appendix E.3
Table 3: Ablations on iTransformer. We replace different components the respective dimension to learn multivariate correlations (Variate) and series representations (Temporal), in addition to component removal. The average results of all predicted lengths are listed here.
ECL.
Traffic
Weather
Solar-Energy
Design
Variate
Temporal
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
iTransformer
Attention
FFN
0.178
0.270
0.428
0.282
0.258
0.278
0.233
0.262
Attention
Attention
0.193
0.293
0,913
0.500
0.255
0.280
0.261
0.291
Replace
FFN
Attention
0.202
0.300
0.863
0.499
0.258
0.283
0.285
0.317
FFN
FFN
0.182
0.287
0.599
0.348
0.248
0.274
0.269
0.287
Attention
w/o
0.189
0.278
0.456
0.306
0.261
0.281
0.258
0.289
w/o
w/o
FFN
0.193
0.276
0.461
0.294
0.265
0.283
0.261
0.283
Analysis of series representations To further validate the claim that feed-forwand networks are more favored to extract the series representations. We conduct representation analysis based on the centered kernel alignment (CKA) similarity (Kornblith et al. 2019). higher CKA indicates more similar representations. For Transformer variants and l'Transformers, we calculate the CKA between the output features of the first and the last block. Notably, previous works have demonstrated that time series forecasting, as low-level generative task, prefers the higher CKA similarity (Wul et al., 2023: Dong et al. 2023) for the better performance. As shown in Figure 7 a clear division line is exhibited, implying that iTransformers have learned more appropriate series representations by inverting the dimension and thus achieve more accurate predictions. The results also advocate inverting Transformer deserves fundamental renovation of the forecasting backbone.
8

---

Published as a conference paper at ICLR 2024
Analysis of multivariate correlations By assigning the duty of multivariate correlation to the attention mechanism, the learned map enjoys enhanced interpretability. We present the case visual- ization series from Solar-Energe in Figure which has distinct correlations in the lookback and future windows. It can be observed that in the shallow attention layer. the learned map shares lots of similarities to the correlations of raw input series. As it dives into deeper layers, the learned map become gradually alike to the correlations of future series, which validates the inverted operation empowers interpretable attention for correlating, and the processes of encoding the past and decoding for the future are essentially conducted in series representations during feed- redd-forwrrding..
Lookback Corretations
Futurre Corretations
0.7
Trandlammr
Infvenar
0.75
iReformer
Reforner
0.5
Flaihforrer
0.50
MSE
0.25
0.3
0.00
-0 25
0.50
0.1
0.75
0.6
0.7
0.8
0.9
1.0
CKA Similarity
00
Score
Figure 7: Analysis of series representations and multivariate correlations. Left: MSE and CKA similarity of representations comparison between Transformers and iTransformers. higher CKA similarity indicates more favored representations for accurate predictions. Right: A case visualization of multivariate correlations raw time series and the learned score maps by inverted self-attention.
Efficient training strategy Due to the quadratic complexity of self-attention, it can be overwhelm- ing for training on numerous variates, which very common in real- world scenarios. In addition to efficient attention mechanisms, we propose a novel training strategy for nigh-dimensional multivariate series by taking advantage previously demonstrated variate generation capability. Concretely, we randomly choose part of the variates in each batch and only train the model with selected variates. Since the number variate channels flexible because of our inverting, the model can predict all the variates for predictions. As shown in Figure g
the performance of our proposed strategy is still comparable with full-variate training, while the memory footprint can be reduced significantly.
ECL
Traffie
Solar-Energy
15
0.4
1000%
80%
61%%
MSE
0.3
0.2
0.1
100%
80%
60%
40%
20%
100%
80%
60%
40%
20%
Sample Ratio
Sample Ratio
Figure 8: Analysis of the efficient training strategy. While the performance (left) remains stable partially trained variates of each batch with different sampled ratios, the memory footprint (right) can be cut off greatly. We provide the comprehensive model efficiency analysis in Appendix D
5 CONCLUSION AND FUTURE WORK
Considering the characteristics of multivariate time series, we propose iTransformer that inverts the structure of Transformer without modifying any native modules. iTransformer regards independent series as variate tokens to capture multivariate correlations by attention and utilize layer normalization and feed-forward networks to learn series representations. Experimentally, iTransformer achieves state-of-the-as performance and exhibits remarkable framework generality supported by promising analysis. In the future, we will explore large-scale pre-training and more time series analysis tasks.

---

Published as a conference paper at ICLR 2024
ETHICS STATEMENT
Our work only focuses on the time series forecasting problem, so there is no potential ethical risk.
7 REPRODUCIBILITY STATEMENT
In the main text, have strictly formalized the model architecture with equations. All the imple- mentation details are included the Appendix, including dataset descriptions, metrics, model, and experiment configurations. The code will be made public once the paper is accepted.
ACKNOWLEDGMENTS
This work was supported by the National Key Research and Development Plan (2021 YFB1715200 the National Natural Science Foundation of China (U2342217 and 62022050), the BNRist Inno- vation Fund (BNR2024RC00010, Ant Group through CCF-Ant Research Fund, and the National Engineering Research Center for Big Data Software.
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. https://arxii.orgp//////6..0.000 607.//.0066....45.......... 2016.
Hinton.
Layer normalization.
Shaojie Bai, Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and
recurrent networks for sequence modeling. arXiv preprint arXiv:1803.01271, 2, 2018.
George EP Box and Gwilym M Jenkins. Some recent advances in forecasting and control. Journal of the Royal Statistical Society. Series C (Applied Statistics), 17(2):91-109, 1968.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. NeurIPS, 2020.
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory- efficient exact attention with io-awareness. NeurIPS, 2022.
Abhimanyu Das, Weihao Kong, Andrew Leach, Rajat Sen, and Rose Yu. Long-term forecasting with tide: Time-series dense encoder. arXiv preprini arXiv:2304.08424, 2023.
Jiaxiang Dong. Haixu Wu, Haoran Zhang, Li Zhang, Jianmin Wang, and Mingsheng Long. Simmtm: simple pre-training framework for masked time-series modeling. arXiv preprint arXiv:2302.00861, 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition scale. ICLR, 2021.
Vijay Ekambaram, Arindam Jati, Nam Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. Tsmixer: Lightweight mlp-mixer model for multivariate time series forecasting. KDD, 2023.
Lu Han, Han-Jia Ye, and De-Chuan Zhan. The capacity and robustness trade-off: Revisiting the chan- nel independent strategy for multivariate time series forecasting. arXiv preprint arX 2023.
Kurt Hornik. Approximation capabilities of multilayer feedforward networks. Neural networks, 4(2): 251-257, 1991.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08880 2020.
10

---

Published as a conference paper at ICLR 2024
Taesung Kim, Jinhee Kim, Yunwon Tae, Cheonbok Park, Jang-Ho Choi, and Jaegul Choo. Reversible instance normalization for accurate time-series forecasting against distribution shift. ICLR, 2021.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ICLR, 2015.
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. JCLR, 2020.
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey E. Hinton. Similarity of neural network representations revisited. ICML, 2019.
Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. SIGIR, 2018.
Jianxin Li, Xiong Hui, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. arxiv: 20/2.07436, 2021.
Zhe Li, Shiyi Qi, Yiduo Li, and Zenglin Xu. Revisiting long-term time series forecasting: An investigation on linear mapping. arXiv preprint arXiv:2305. 2023.
Minhao Liu, Ailing Zeng, Muxi Chen, Zhijian Xu, Qiuxia Lai, Lingna Ma, and Qiang Xu. Scinet: time series modeling and forecasting with sample convolution and interaction. NeurIPS, 2022a.
Yong Liu, Haixu Wu, Jianmin Wang, and Mingsheng Long. Non-stationary transformers: Rethinking the stationarity in time series forecasting. NewrIPS, 2022b.
Yong Liu, Chenyu Li, Jianmin Wang, and Mingsheng Long, Koopa: Learning non-stationary time series dynamics with koopman predictors. arXiv preprint arXiv:2305.18803 2023.
Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. time series is worth 64 words: Long-term forecasting with transformers. ICLR, 2023.
Boris Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-BEATS: Neural basis expansion analysis for interpretable time series forecasting. ICLR, 2019.
Adam Paszke, S. Gross, Francisco Massa, A. Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Z. Lin, N. Gimelshein, L. Antiga, Alban Desmaison, Andreas Köpf. Edward Yang, Zach De Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. NeurlPS, 2019.
Syama Sundar Rangapuram, Matthias W Seeger, Jan Gasthaus, Lorenzo Stella, Yuyang Wang, and Tim Januschowski. Deep state space models for time series forecasting. NewrIPS, 2018
David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Deepar: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting, 36(3): 1181-1191, 2020.
Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lueas Beyer, Xiaohua Zhai, Thomas Un- terthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mlp-mixer: An all-mlp architecture for vision. NeurIPS, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. NeurlPS, 2017.
Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long, Autoformer: Decomposition transformers with Auto-Correlation for long-term series forecasting. NeurIPS, 2021.
Haixu Wu, Jialong Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Flowformer: Linearizing transformers with conservation flows. ICML, 2022.
Haixu WU, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. Timesnet: Temporal 2d-variation modeling for general time series analysis. ICLR, 2023.

---

Published as a conference paper at ICLR 2024
Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? AAAI, 2023.
Yunhao Zhang and Junchi Yan. Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting. ICLR, 2023.
Zheng Zhao, Weihai Chen, Xingming Wu, Peter CY Chen, and Jingmeng Liu, Lstm network: a deep learning approach for short-term traffic forecast. IET Intelligent Transport Systems, 11(2):68-75, 2017.
Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. FEDformer: Frequency enhanced decomposed transformer for long-term series forecasting. ICML, 2022.
12

---

Published as a conference paper at ICLR 2024
A IMPLEMENTATION DETAILS
A.1 DATASET DESCRIPTIONS
We conduct experiments 7 real-world datasets to evaluate the performance of the proposed iTransformer including (1) ETT et al. 2021) contains 7 factors of electricity transformer from July 2016 to July 2018. There are four subsets where ETTh1 and ETTh2 are recorded every hour, and ETTml and ETTm2 are recorded every 15 minutes. (2) Exchange (Wu et al.) (222)) collects the panel data of daily exchange rates from 8 countries from 1990 2016. (3) Weather (Wu al. 2021) includes 21 meteorological factors collected every 10 minutes from the Weather Station of the Max Planck Biogeochemistry Institute in 2020. (4) ECL (Wu al. 202T) records the hourly electricity consumption data of 321 clients. (5) Traffic (Wu et al. (021) collects hourly road occupancy rates measured by 862 sensors of San Francisco Bay area freeways from January 2015 to December 2016. (6) Colar-Energy (Lai et al. 2018) records the solar power production of 137 PV plants in 2006, which are sampled every 10 minutes. (7) PEMS contains the public traffic network data California collected by 5-minute windows. We use the same four public subsets (PEMS03, PEMS04, PEMS07, PEMS08) adopted in SCINet (Liu et al. 2022a).
Apart from the public datasets widely used forecasting benchmarks, we also collect set of Market datasets of a real-world application, which records the minute-sampled server load of Alipay online transactions between January 30th, 2023, and April 9th, 2023 with the number of variates varied from 285 to 759. It includes 6 sub-datasets, which are divided according to diverse transaction domains.
We follow the same data processing and train nain-validation-test set split protocol used in TimesNet (Wu et al. 2023), where the train, validation, and test datasets are strictly divided according to chronologi- cal order to make sure there are data leakage issues. As for the forecasting settings, fix the length of the lookback series as 96 in ETT, Weather, ECL, Solar-Energy, PEMS, and Traffic, and the prediction length varies in (96, 192, 336, 720). For the PEMS dataset, the prediction length varies in {12, 24, 36, 48), which is the same as SCINet, the previous state-of-the tf-th-ra-t- on this dataset. For the Market dataset, the lookback contains the past one day observations with 144 time points and the forecasting length varies in {12, 24, 72, 144). The details of datasets are provided in Table
A.2 IMPLEMENTATION DETAILS
Algorithm 1 i'Transformer Overall Architecture.
Require: Input lookback time series X E TRTN;; input Length T; predicted length S; variates number N; token dimension D; iTransformer block number L.
X X.transpose
E NXT
2: & Multi-layer Perceptron works on the last dimension to embed series into variate tokens.
3: H ML.P(X)
H E N
4: for in (I,..., L):
Run through iTransformer blocks.
5: Self-attention layer is applied on variate tokens.
6: H'-1 LayerNorm H + Self-Attn(H'-))
> HH e NxD
7: Feed-forward network is utilized for series representations, broadcasting each token.
8: H' LayerNorm H + Feed (onnddFooward(HH"))
D H N
9:
LayerNorm is adopted on series representations to reduce variates discrepancies.
10: End for
Y MLP (H
Project tokens back to predicted series, Y E Nx5
12: & = Y.transpose
> > E SXN
13: Return Y
Return the prediction result Ý
13

---

Published as a conference paper at ICLR 2024
Table 4: Detailed dataset descriptions. Dim denotes the variate number of each dataset. Dataset
Size denotes the total number of time points in (Train, Validation, Test) split respectively. Prediction Length denotes the future time points be predicted and four prediction settings are included in each dataset. Frequency denotes the sampling interval of time points.
Dataset
Dim
Prediction Length
Dataset Size
Frequency
Information
ETTh1, ETTh2
7
(96, 192, 336, 720)
(8545, 2881, 2881)
Hourly
Electricity
ETTml, ETTm2
7
(96, 192, 336, 720)
(34465, 11521, 11521)
15min
Electricity
Exchange
8
(96, 192, 336, 720)
(5120, 665, 1422)
Daily
Economy
Weather
21
(96, 192, 336, 720)
(36792, 5271, 10540)
10min
Weather
ECL
321
(96, 192, 336, 720)
(18317, 2633, 5261)
Hourly
Electricity
Traffic
862
(96. 192, 336, 720)
(12185, 1757, 3509)
Hourly
Transportation
Solar-Energy
137
196. 192, 336, 720)
(36601, 5161, 10417)
10min
Energy
PEMS03
358
{12, 24, 48, 96)
(15617, 5135, 5135)
5min
Transportation
PEMS04
307
[12, 24, 48, 96)
(10172, 3375, 3375)
5min
Transportation
PEMS07
883
{12, 24, 48, 96)
(16911, 5622, 5622)
5min
Transportation
PEMS08
170
[12, 24, 48, 96)
(10690, 3548, 3548)
5min
Transportation
Market- Market-Merchant
285
{12, 24. 72, 1443
(7045, 1429, 1429)
10min
Transaction
Market-Weally
485
(12.24. 72. 1443
(7045, 1429, 1429)
10min
Transaction
Market-Finance
405
(12, 24, 72, 144)
(7045, 1429, 1429)
10min
Transaction
Market-Terming
307
(12, 24, 72, 144)
(7045, 1429, 1429)
10min
Transaction
Market-Payment
759
{12, 24, 72, 144)
(7045, 1429, 1429)
10min
Transaction
395
{12, 24, 72, 144)
(7045, 1429, 1429)
10min
Transaction
All the experiments are implemented in PyTorch (Paszke et al. 2019) and conducted on a single NVIDIA P100 16GB GPU. We utilize ADAM (Kingma & Ba 2015) with an initial learning rate in {10-3,5 3 5 x 10 10 11044 and L2 loss for the model optimization. The batch size is uniformly set to 32 and the number of training epochs is fixed 10. We set the number of inverted Transformer blocks in our proposed model L, E {2, 3, 43. The dimension of series representations D is set from {256, 512). All the compared baseline models that we reproduced are implemented based on the benchmark of TimesNet (Wu et al. 2023) Repository, which is fairly built on the configurations provided by each model's 's original paper or official code. We provide the pseudo-code of iTransformer in Algorithm We also report the standard deviation of ITransformer performance under five runs with different random seeds in Table which exhibits that the performance of iTransformer stable.
Table 5: Robustness of iTransformer performance. The results are obtained from five random seeds.
Dataset
ECL
ETTh2
Exchange
Horizon
MSE
MAE
MSE
MAE
MSE
MAE
96
0.048
0.240
0.297=0.002
0.349=0.001
0.088=00001
0.209+0.001
192
0.162-00.02
0.253+0.002
0.380=00001
0.400=0.001
0.181+0.001
0.304+0.001
336
0.178-10.000
0.269
0.428=0.002
0.432=0.001
0.334+0.001
0.419+0.001
720
0.225_10.00
0.31710.007
1..00
0,445
0.82910.012
0.69110.005
Dataset
Solar-Energy
Traffic
Weather
Horizon
MSE
MAE
MSE
MAE
MSE
MAE
96
0.203-40.002
0.237=0.002
0.395=00000
0.268=0.001
0.174+0.000
0.214+0.000
192
0.233-10.002
0.261100001
0.417=0.002
0.276=0.001
1.0002
0.25410.001
336
0.208
0.273
0.433=0.004
0.283=0.000
0.27811.002
0.296100001
720
0.249400001
0.27540.000
0.467=0.003
0.302-0.000
0.35810.000
0.34940.000
14

---

Published as a conference paper at ICLR 2024
B ABLATION STUDIES
To claborate on the rational business of Transformer components, we conduct detailed ablations covering replacing components (Replace) and removing components (wo).. Since the average results are listed in Table due to the paper limit, we provide detailed results and analysis here.
As shown in Table among various architectural designs, iTrrnsformer generally exhibits superior performance, which learns multivariate correlations by self-attention and encodes series represen- tations by FFN. Nevertheless, the arrangement of the vanilla Transformer can lead degenerated performance, indicating the misuse of Transformer components on the time series modality. Based on the relatively poor results of the second (both attentions) and the third (the vanilla Transformer) designs, one of the reasons for that may lie in the attention module over the temporal tokens of the lagged time series, which we elaborate more with the datasets support Section
It is also notable that applying FFN on both dimensions can also lead to fair performance on datasets with small variate numbers (such as Weather with 21 variates). Still, with the increasing of variate numbers in challenging multivariate forecasting tasks, the importance of capturing multivariate corre- lations is ever more highlighted. We note that the heterogeneity of variates can be hardly considered by the vanilla Transformer. During embedding, the variates projected into indistinguishable channels, which ignores the inconsistent physical measurements and thus fails to maintain the in- dependence of variates, let alone capture and utilize the multivariate correlation. Consequently, by incorporating the advanced attention module for the variate correlating, the first (iTransformer) and the fifth (attention on variates) designs perform more effectively in challenging multivariate datasets.
In a nutshell, both temporal dependencies and multivariate correlations are of importance for mul- tivariate time series forecasting. The proposed i'Transformer employing the self-attention module to disentangle the correlations between variate tokens proves to be more powerful and interpretable than feed-forward networks, thereby further boosting the performance on challenging multivariate datasets and enhancing the model capacity.
C HYPERPARAMETER SENSITIVITY
We evaluate the hyperparameter sensitivity of iTransformer with respect to the following factors: the learning rate ir, the number of Transformer blocks L, and the hidden dimension D of variate tokens. The results are shown in Figure 9 We find that the learning rate, as the most common influencing factor, should be carefully selected when the number of variates is large (ECL, Traffic). The block number and hidden dimension are not essentially favored to be as large as possible in iTransformer.
Learning Rate
Block Number
Hidden Dimension
0.5
0.45
0.45
0.40
0.4
0.35
0.35
MSB
0.30
0.30
0.25
0.25
0.2
0.20
0.20
0.15
0.001
0.0005
0.0003
0.0001
256
512
1024
ETT
ECL
Traffic
Weather
Figure 9: Hyperparameter sensitivity with respect to the learning rate, the number of Transformer blocks, and the hidden dimension of variate tokens. The results are recorded with the lookback window length T 96 and the forecast window length S 96.
D MODEL EFFICIENCY
We comprehensively compare the forecasting performance, training speed, and memory footprint of the following models: iTransformer, iTransformer with our efficient training strategy and iTransformer
15

---

Published as a conference paper at ICLR 2024
Table 6: Full results of the ablation on iTransformer. We apply different components on the respective dimension learn multivariate correlations (Variate) and series representations (Temporal), in addition to removing the specific component of Transformer.
Prediction
ECL.
Traffic
Weather
Solar-Energy
Design
Variate
Temporal
Lengths
MSE
MAB
MSE
MAE
MSE
MAE
MSE
MAE
96
0.148 0.240
0.395 0.268
0.174
0.214
0.203
0.237
192
0.162 0.253
0.417 0.276
0.221
0.254
0.233
0.261
iTransformer
Attention
FFN
336
0.178
0.269
0.433
0.283
0.278
0.296
0.248
0.273
720
0.225
0.317
0.467
0.302
0.358
0.349
0.249
0.275
Avg
0.178 0.270
0.428
0.282
0.258
0.279
0.233
0.262
96
0.161
0.263
1.021
0.581
0.168
0.213
0.227
0.270
192
0.180
0.280
0.834
0.447
0,217
0.256
0.255
0.292
Attention
Attention
336
0.194
0.296 0.906
0.493
0.277
0.299
0.279
0.301
720
0,238
0.331
0.892
0.477
0.356
0.351
0.283
0.300
Avg
0.193
0.293
0.913
0.500
0.255
0.280
0.261
0.291
96
0.169
0.270
0.907
0.540
0.176
0.221
0.247
0.299
Replace
192
0.189
0.292
0.839
0.489
0.224
0.261
0.275
0.305
FFN
Attention
336
0.204
0.304
0.248
0.364
0.279
0.301
0.317
0.337
720
0.245
0.335
1.059
0.606
0.354
0.347
0.301
0.329
Avg
0.202
0.300
0.863
0.499
0.258
0,283
0.285
0.317
0.159
0.261
0.606
0.342
0.162
0.207
0.237
0.277
192
0.171
0.271
0.559
0.342
0.211
0,252
0.273
0.293
FFN
FFN
336
0.187
0.287
0.569
0.348
0.270
0.293
0.284
0.287
720
0.211
0.307
0.664
0.359
0.349
0.345
0.284
0.289
Avg
0.182
0.287
0.599
0.348
0.248
0,274
0.269
0.287
96
0.163
0.254
0.427
0.296
0.177
0.219
0.226
0,266
192
0.174
0.263
0.446
0.300
0.226
0.259
0.255
0.288
Attention
w/o
336
0.191
0.280
0.459
0.306
0.281
0.298
0.275
0.301
720
0.228
0,315
0.492
0.324
0.359
0.249
0.275
0.301
w/o
Avg
0.189
0.278
0.456
0.306
0.261
0.281
0.258
0.289
96
0.169
0.253
0.437
0.283
0.183
0.220
0.228
0.263
192
0.177
0.261
0.449
0.287
0.231
0.262
0.261
0.283
w/o
FFN
336
0.194
0.278
0.464
0.294
0,285
0.300
0.279
0.294
720
0,233
0,311
0.496
0.313
0.362
0.350
0.276
0.291
Avg
0.193
0.276
0.461
0.294
0.265
0.283
0.261
0.283
Traffic (862 Varictes]
Weather Variates)
0.9
Memory Footprint
0.8
TIDE
0 0.93G3
LIDGB
Tronsformer
0.
PatchTST A35GR 435ex.
Memory Footprint
DLinsor
Troneformer
Düsser
1145K, tahen
iFlowformer
0.6
MAGA, Pire
Treesformen (STicien)
0.4
[Transformer 25009, 26iwa
Cressformer
ITrondorias ooeOx,
Crossfommr
Troofurier
PatchTST
0.
753
Training Time (ms/iter)
Training Time [mx/ter]
Figure 10: Model efficiency comparison under input-96-predict-96 of Weather and Traffic.
16

---

Published as a conference paper at ICLR 2024
with the efficient flow attention module (Wu et al. 2022; linear models: DLinear Zeng al. 2023) and TiDE (Das et al. 2023); Transformers: Transformer Vaswani et al. 2017 PatchTST (Nie eet al., 2023), and Crossformer (Zhang & Yan 2023). The results recorded with the official model configuration and the same batch size. In Figure 10, we compare the efficiency under two representative datasets (21 variates in Weather and 862 in Traffie) with 96 time steps for lookback.
In a nutshell, the efficiency of iTrrnsformeer exceeds other Transformers in datasets with a relatively small number of variates (Weather). In datasets with numerous variates (Traffic), the memory footprints are basically the same as Transformers variates, but iTransformer can be trained faster. Based on the complexity of (NV2) of the attention module, where N is the number of tokens, Transformer surpasses i'Tranformer on efficiency in this case because of N = 96 for the temporal token and N 862 for the variate token. Meanwhile, iTransformer achieves better performance on numerous variates, since the multivariate correlations can be explicitly utilized. By adopting limear-complexity attention (Wu et al.) 2022) or the proposed efficient training strategy as mentioned in Figure 8 (trained on 20% variates and forecast all variates), iTransformer can enjoy a comparable speed and memory footprint with linear models. Also, the two strategies can be adopted together.
E SHOWCASES
E.1 VISUALIZATION OF MULTIVARIATE CORRELATIONS
By using the attention mechanism on variate tokens, the resulting learned map becomes more interpretable. To present an intuitive understanding of the multivariate correlations, we provide three randomly chosen case visualizations of the time series from Volar-Energy in Figure We provide the Pearson Correlation coefficients of each variate of the raw series by the following equation:
E, - (U 9) Pay ET
V. PV g)
where z,, Vi E R run through all time points of the paired variates to be correlated. All the cases have distinct multivariate correlations in the lookback and forecast window because the dataset exhibits obvious seasonal changes in the daytime and night. On the second row each case, we provide the learned pre-Softmax maps of the self-attention module in both the first and the last layers. As observe in the shallow attention layer (left), we find that the learned map is similar to the correlations of the raw lookback series. As go deeper into the layers (right), the learned map gradually becomes more similar to the correlations of the future series to be predicted. This demonstrates that the inverted operation allows for interpretable attention in correlating, and that encoding of the past and decoding for the future are conducted through series representations during layer stacking.
Lookbtuck Corretatieno
Faluarc Corvchatless
Locksbock Derchitous
Putare Corrdation
Looklauck Correlations
Piture Correlatios
1.00
0.00
0.00
0.50
0.35
-120
Soore Map Lager
Score Map of Lager
Score Map Layer
Scare Map of Layer
Soore Map of Layer
Scure Map Layer
Figure 11: Multivariate correlations of the lookback series and future series and the learned score maps by inverted self-attention of different layers. Cases all come from the Voler-Energy dataset.
We present another interesting observation in Figure 12 to show that the attention module of iTrans- former has enhanced interpretability. We provide randomly chosen multivariate time series from Market. In this dataset, each variate represents the monitored values of a service interface of kind,
17

---

Published as a conference paper at ICLR 2024
and the service can be further grouped into refined application categories. We divide these variates into corresponding applications (as listed on the top bar App), such that adjacent variates belong to the same application and we reveal the application index by the top bar.
We visualize the time series the variates and plot the learned multivariate correlations with the marks of specific correlations between variates. On the one hand, observe clear partitioning in the multivariate correlations map, indicating the grouping of variates. On the one hand, the marked correlation values can reflect the correlation the raw series, where the similarity of variates from the same application becomes closer than the pairs from the different groups. Therefore, highly correlated variate will be leveraged for the next interaction and thus benefit for multivariate forecasting.
Case
Case
Vasiar
305
165
baicher App
Yasiato App
1340
125
162
Valae Aa
257
Figure 12: Visualization of the variates from the Market dataset and the learned multivariate correla- tions. Each variate represents the monitored interface values of an application, and the applications can be further grouped into refined categories. The color bar is shared with Figure
E.2 VISUALIZATION OF PREDICTION RESULTS
To provide clear comparison among different models, we list supplementary prediction showcases of four representative datasets in Figures 13 16 which are given by the following models: i'Transforrer,
PatchTST (Nie et al. 2023). DLinear ZZng et al., (023). Crossformer (Zhang & Yan (023).
Autoformer (Wu et al. 2021] Transformer (Vaswani et al. 2017). Among the various models,
iTransformer predicts the most precise future series variations and exhibits superior performance.
iTransformer
PatchTST
DLinear
119
Crossformer
Transformer
Figure 13: Visualization of input- 96- 90---6---c---0000 96 results the Traffic dataset.
E.3 RISKS OF EMBEDDING MULTIVARIATE POINTS OF A TIMESTAMP
As aforementioned,, the embedding approach of the previous Transformer fuses multiple variates representing potentially delayed events and distinct physical measurements, which may fail to learn variate-centric representations and result in meaningless attention maps. We provide the visualization
18

---

Published as a conference paper at ICLR 2024
iTransformer
PatchTST
DLinear
04-
0.3
Crossformer
Autoformer
Transformer
-15
Figure 14: Visualization of input- 96 predict- 96 results on the ECL dataset.
Trronsoormer
034
08:
0.4-
Crossformer
Autoformer
Transformer
0.4
1.10
0.2
Figure 15: Visualization of input-900 19- 96---6ddcc--9999 96 results on the Weather dataset.
iTransformer
PatchTST
DLinear
3.0
Vig
-00
2/0
Crossformer
Autoformer
Transformer
Figure 16: Visualization of input-96 111.-06---------. 96 results on the PEMS dataset.
19

---

Published as a conference paper at ICLR 2024
case of Traffic (Liu al. 2022a which is collected from sensors Los Angeles city roads in different areas, As shown mn Figure 17 we can observe a strong correlation between the multivariate time series of the dataset, while they also exhibit obvious phase offset, which is due the systematical time lags in the road occupancy that each series describes. Since the sensors are installed different areas of the highway, event (such traffic jam) can affect road occupancy with different delays.
Road occupancy
Road
Sensor
Sensor 858
Sensor 859
Sensor U61
Sensor 860
Semwor Senser 4
0.12
0.00
6:00
12:00
Figure 17: Visualization of partial variates of Traffic. We can observe that several series exhibit strong synchronization (such as Sensor 2 and Sensor 4), and there also exist obvious delays and advances between series (such as Sensor and Sensor 2, Sensor 859 and Sensor 861).
Besides, we observe the significantly declined performance on the second and third designs of Traffic in Table which apply attention temporal tokens. In our opinion, capturing temporal dependencies by attention is not big problem. But it is based on the fact that the time points of each timestamp essentially reflect the same event to enclose semantic representation. Since there inherent delays between the time points, the performance can degrade a lot because of the meaningless attention map, unless the model has enlarged respective field to learn about the decay causal process.
Other risks can be aroused from the distinct variate measurements, such as organizing together different meteorological indicators (the temperature and rainfall) in the Weather dataset (Wu et al.) 2021). and the quantity and proportion of the same observation in ILI (Wu et al. (023). Given these potential risks, iTTranformeer proposes a new paradigm that embeds the whole series as the variate token, which can be more robust to extensive real- wolld scenarios, such delayed events, inconsistent measurements, irregular (unevenly spaced) time series, systematical delay of monitors. and the time interval of generating and recording different time series.
F
FULL RESULTS
FULL PROMOTION RESULTS
We compare the performance of Transformer and iTransformer on all datasets in Table 7 Consistent and great promotions can be achieved, indicating that the attention and feed-forward network on the inverted dimensions greatly empower Transformers in multivariate time series forecasting, leaving an instructive direction to build up the foundation model of extensive time series data.
Table 7: Full performance comparison between the vanilla Transformer and the proposed iTransformer. The results are averaged from all four prediction lengths.
ETT
PEMS
Solar-Energy
Traffic
Weather
Datasets Metric
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
Transformer
2,750
1.375
0.277
0.372
0.157
0.263
0,256
0.276
0.665
0.363
0.657
0.572
iTransformer
0.383
0.407
0.178
0.270
0.113
0.221
0.233
0.262
0.428
0.282
0.258
0.279
Promotion
86.1%
70.4%
35.6%
27.4%
28.0%
16.0%
9.0%
5.1%
35.6%
22.3%
60.2%
50.8%
F.2 FULL FRAMEWORK GENERALITY RESULTS
We apply the proposed inverting framework to Transformer and its variants: Transformer Vaawwani
et al. 2017). Reformer (Kitaev al. 2020), Informer (Li et al. 202T), Flowformer (Wu al.
20

---

Published as a conference paper at ICLR 2024
2022). Flashformer (Dao et al. 2022). The averaged results are shown in Table due to the limited pages. We provide the supplementary forecasting results in Table 8 The results demonstrate that our Transformers framework can consistently promote these Transformer variants, and take advantage of the booming efficient attention mechanisms.
Table Full results of Transformers with our inverted framework. Flashformer means Transformer equipped with the nardware-accelerate FlashAttention (Dao et al. 2022).
Transformer
Reformer
Informer
Flowformer
Flashformer
Models
(2017)
2020)
202T
2022
2022
Metric
MSE
MAE
MSE
MAE
MSE
MAB
MSE
MAE
MSE
MAE
96
0.260
0.358
0.312
0.402
0.274
0.368
0.215
0.320
0.259
0.357
192
0.266
0.367
0.348
0.433
0.296
0.386
0.259
0.355
0.274
0.374
Original
336
0,280
0.375
0.350
0.433
0,300
0.394
0.296
0,383
0.310
0.396
720
0.302
0.386
0.340
0.420
0.373
0.439
0.296
0.380
0.298
0.383
ECL
Avg
0.277
0.372
0.338
0.422
0.311
0.397
0.267
0.359
0.285
0.377
96
0.148
0.240
0.182
0.275
0.190
0.286
0.183
0.267
0.178
0.265
192
0.162
0.253
0.192
0.286
0.201
0.297
0.192
0.277
0.189
0.276
+nverted
336
0.178
0.269
0,210
0.304
0.218
0.315
0.210
0.295
0.207
0.294
720
0.225
0.317
0.249
0.339
0.255
0.347
0.255
0.332
0.251
0.329
Avg
0.178
0.270
0.208
0.301
0.216
0.311
0.210
0.293
0.206
0.291
96
0.647
0.357
0.732
0.423
0.719
0.391
0.691
0.393
0.641
0.348
192
0.649
0.356
0.733
0.420
0.696
0.379
0.729
0.419
0.648
0.358
Original
336
0.667
0.364
0.742
0.420
0.777
0.420
0.756
0.423
0.670
0.364
720
0.697
0.376
0.755
0.432
0.864
0.472
0.825
0.449
0.673
0.354
Traffic
Avg
0.665
0.363
0.741
0.422
0.764
0.416
0.750
0.421
0.658
0.356
96
0.395
0.268
0.617
0.356
0.632
0.367
0.493
0.339
0.464
0.320
192
0.417
0.276
0.629
0.361
0.641
0.370
0.506
0.345
0.479
0.326
+Inertedd
336
0.433
0.283
0.648
0.370
0.663
0.379
0.526
0.355
0.501
0.337
720
0.467
0.302
0.694
0.394
0.713
0.405
0.572
0.381
0.524
0.350
Avg
0.428
0.282
0.647
0.370
0.662
0.380
0.524
0.355
0.492
0.333
96
0.395
0.427
0.689
0.596
0.300
0.384
0.182
0.233
0.388
0.425
192
0.619
0.560
0.752
0.638
0.598
0.544
0.250
0.288
0.619
0.560
Original
336
0.689
0.594
0.639
0.596
0.578
0.523
0.309
0.329
0.698
0.600
720
0.926
0.710
1.130
0.792
1.059
0.741
0.404
0.385
0.930
0.711
Weather
Avg
0.657
0.572
0.803
0.656
0.634
0.548
0.286
0.308
0.659
0.574
96
0.174
0.214
0.169
0.225
0.180
0.251
0.183
0.223
0.177
0.218
192
0.221
0.254
0.213
0.265
0.244
0.318
0.231
0.262
0.229
0.261
+nverted
336
0.278
0.296
0.268
0.317
0.282
0.343
0.286
0.301
0.283
0.300
720
0.358
0.349
0.340
0.361
0.377
0.409
0.363
0.352
0.359
0.251
Avg
0.258
0.279
0.248
0.292
0.271
0.330
0.266
0.285
0.262
0.282
F.3 FULL RESULTS OF VARIATE GENERALIZATION
We divide the variates of each dataset into five folders, train models with only 20% of variates of one folder, and directly forecast all variates without fine-tuning. We adopt two strategies for Transformers to generalize on unseen variates: (1) CI- CI-Transformers (Nie et al., 2023): Channel Independence regards each variate of time series independent channels, and trains with shared backbone. During inference, the model predicts variates one by one, but the procedure can be time-consuming. (2) iTransformers: with the flexibility of the attention mechanism that the number of input tokens can be dynamically changeable, the amount of variates as tokens no longer restricted and thus feasible to vary from training and inference, and can
allow the model to be trained on arbitrary variates.
As shown in Table I8 Transformmrs can be naturally trained with 20% variates and accomplish forecast on all variates with the ability to learn transferable representations.
21

---

Published as a conference paper at ICLR 2024
Solar-Encrg
0.7
1.0
3.32
8.5
0.8
0.20-
10076
0.24
-1004
1.374
6.4
0.10-
0.1
0.05-
PRMSI1
PFMSIN
PEMSOS
0.150
0.150
0.250-
0.125
0.125
360
0.000
10025
1390
0.190
914
0.130
0,800-
HILD
6625
6825
6400
Tranbener
Inforte:
Reform:
Flovfame
6:00
Iafore:
Reform::
Flowkmme
6,000
Tammbeenct
laferter
Reforme
Flowfame
GI Toneformrss (100% vanano
CI Toanttomees (20% vanano
IThansDemmes (109/5 varata)
Mersshrmess vantan))
Figure 18: Full performance of generalization CI-Transfomers. We divide the variates of each dataset into five folders, train with 20% variates, and use the trained model to forecast all varieties. We plot the averaged results of all five folders.
unseen variates, comparing the iTransformers with
F.4 FULL FORECASTING RESULTTS
The full multivariate forecasting results provided in the following section due to the space limita- tion of the main text. We extensively evaluate competitive counterparts on challenging forecasting tasks. Table contains the forecasting results on the four public subsets from PEMS (Liu al. 2022a). Table contains the detailed results of all prediction lengths of the nine well-acknowledgee forecasting benchmarks. And Table 11 records the Market results for Alipay server load forecasting. The proposed model achieves comprehensive state-of-the-art in real-world forecasting applications.
Table 9: Full results of the PEMS forecasting task. We compare extensive competitive models under different prediction lengths following the setting of SCINet (2022a). The input length is set to 96 for all baselines. Avg means the average results from all four prediction lengths.
Models
Trannformer (Ours)
Riiin
PatchTST
Cinsformer
TDD
TimesNet
DLinear
SCINet
PRDformer
Stationary
Autoformer
2023
2023
202
2023
2023
20225
2021
Metric
MSE
MAB
MSE
MAE MSE MAE MAEMSE MSE MAE
MSE MAE MAE MSE MAE MSE MAE MSE MAE MSE MAE
MSE MAE
PEMS03
0.071
10.16
23000.0999 0.216/0.99
1.178 30s 0.192/0. 0.243 0.066 0.172 0.126 0.251
0.081 D. 0.18 0.385
0.095
0.201
0.246
0.334 0.142 0.25
0.240 0.257 0.371 18 0. 0.223 201 0.317 0.085 0.198 0.149 0.275 0.105 0.2140..333 0.334 0.440
0.125
0.236
005511
0.529 0.319
0.379
0.155 0.260
0.425 0,127 0.238
0.348
154 0.257
0.782
0.164
0.275
1.057 0.787
(1.669 0.370
0.367 0.490 0.539
0..47 0.515 0.178 28) 0
348
0.434
0.247
0.336 1.031 0.796
Avrg
00.113
0.221
10.495
0.472|0.180
0.291
16.
0.281 0.326 0.419/0.147
0.248/0.277
0.375
0.114 0.224 0.213 0.327 0.147 0.249/0.667 0.601
PEMS04
0.078
0.183
10.138
0.252
10.
0.221
10.098
0.218 0.219 0.340|0.087
0.195
0.272 0.073 0.177 0.138 0.262 0.088 0.196/0.121 0.191
0.005
0. 0.205
0.258
1.13 0.
0.348
0,275 01131
0.256
0.292 0.398 103
0.21550.223
0,340
0..184 0.193 0.177 0.293 104 0. 0.216 0.449 0.509
0.120
0.233
0.572
0.544 0.229
0.339 0.339 0.205 0.326 0.109 0.136
0.136
0.355
0.137
0.099 0.211 0.270 0.368 0.137 0.25] 0.646 0.610
0.150
0.262
1.137
0.820
0322000.200
0.291 0.389
0.457 0.492 0.532 1.190 0.303 (0.3004552240 0.504
0.114 0.227 0.341 0.427 181 0.297 00912 0.748
Avg
10.111
0.221
00.526
0.491
|o. 195
0.307 (0.099
0.314 o 0.353 0.437/0.1 12 0.241 (0.295 0.388 0.192 0.202 0.231 O. 0.337 0. 127 (30000661000000 0.590
PEMS07
12
00067
0.165
18
0.235
0.005 0.207 (0094 0.200 0.173 o jo. 082 0.181 (0. 0.242 0.068 0.171 0.109 0 0.225 1.185 0.336
0.088
0.199
0.3342
0.341
0.139 0.347 0:271 0.383 0. 101 0.204 0.210 0.329 0.119
0.344 0.102
0.420
48
0.110
0.215
0.562
0.541
(0.253
0.340
0.311
0.369
0.456
0.495
0. 0. 0.238 0.458 0.149 0.237
0.165 0.288 131 0.240 0.390 0.430
0.139
1.096 0.795
0.346 0.404
0.396
0.628 0.577 0.181 0.279/0.594 0.553
0.262 0.376 0.187 0. 0.554 0.578
Avg
0.101
|0.501 0.478|0.211
0.303
|0.235
0.380
0.4-1j0.122 0.225/0.329 0.395
9.119
191199999 0.234
0.283 0../0012222 0.230 0.151
0.079
10.133 0.247/0.166
0.232
0.214 1 02227 0.343/0.112
0.212 0.276
0.087
0.184
0.273
2.3077
0.485
PEMS08
24
0.115
0.219
0.249 0.343 0.224
0.281
0.215
0,260
0.318 0.109 0.141 0.149..114
0.248
0.353
0.122
0.221
0.210
0.301 0.
0.236 0.236...16 0.502
0.569
0.544
0.354
0.355
0.497 0.510 1999
0.211 0.294
0.733
96
0.221
0.267
1. 166 0.814
[1.408 0..47
(1.377
0.307
0.721 0.592
320
0.351
(0.674
0.565
(1.236
0.300
0.442
0.463
(. 0.345 0.367
1 1.385 0.915
Avgg
00.150
0.226
[0.529
0.487
(.57)..2000
0.321
10.268
0.307 0.441 0.464/0.199 193 0.27 (0.37 0.416 0.158 0.244 0.286 0.358/0.201 0.201 0.277/0..814 0.659
Count
0
0
0
0
0
0
0
0
22

---

Published as a conference paper at ICLR 2024
Table 10: Full results of the long-term forecasting task. We compare extensive competitive models under different prediction lengths following the setting of TimesNet (2023). The input sequence length is set to 96 for all baselines. Aug means the average results from all four prediction lengths.
Models
Transfommrr
RLinear
PatcTTST
Crossformer
TIDE
TimesNet
DLinear
SCINet
PEDformer
Stationary
Autoformer
(Ours)
2023
2023
2023
2023
2023
2023
2022a
A022
20226
2021
Metric
MSB
MAB
MSE
MAE
MSE MAE]
MSE
MSE
MAE MSE
MAE
MSE
MAE
MSE MAE
MSE MAE
MSE MAE MSE MAE
MSE MAB
96
0.33d
0.368
0.335
0.399 0.367 00.404 0.426 0.364 0.387 0.375 0.372 0.418 0.438 0,379 0.419)0.386 0.398/0.505 0.475
ETTml
192
0.377
0.391
0.391
0.392 0.367 0.385 01,150 0.151 0.398 0,401 0.374
0.387
0.389 0.439 0,450 0.426 0.M11 0,459 0.144 (05553 0.196
336
0.426
0.420
0.424
0.415 0.399
0.530 0.515 04428
0.410
0:411
(0:413
0.413 0.490 0,485 o 445 0.459 0.495 0.464 (0.63) 0.537
720
0.191
0.159
[0:187
0.45500.454 0.439
0.439/0.666 (0666 0.589 0.487
0:461
0.161
0:117
0.450
0711 0.453 0.595 0.550 0.543 0.190 0.516/0.671 0.561
Avg
00.407
0.410
[0.414
0,407
00.37
0.400/0.513 0.496 0.419 0.419|0.400
0.40660.4403 0.407 00.485 0.481 0.448 0.481 0.456/0.588 0.517
ETTh2 ETTHL ETTm2
96
10.180
0.264
10. 18.182
0.265/p
0.259 00
1.175
0.366 0.2077 0 305/0. 187 D. 0.267/0.198 0.292 0.2286 0.377 O 0.203 0.287 192 0. 0.277 (02255 0.339
0.274/0.255
192
0.250
0.309
0.246
0.304
0.241
0.444
0.492 0,290 0.364
0.309 0.362 0.399 0,445 0.269 0.328 0.280 0.339/0.28 0.288 0.340
336
0.311
0.348
0.307
0,342
0.305
0.343 (1597 0,542 0.377 0.422 0.321 0.351 (0.369 0.427 0.637 0,591 0.325 0.366 0.334
0.261 0.339 0,372
720
0.412
0.407
0
0.407
0.39%
0.39%
0..40 0.400 1,730
1.042 0,558 0.524
0.524|0.408 0..00 0.403 (0554 0.522 0.960
0.421 0.415 0.413 0.432
Aug
002288
0.332
|0 10.286
0.327
0.281 0.326/0.757
0.610 0.358 0.404|0.29) 0.333/0.350 0.401 0.571 0.537 o 0.349
00..9/00.0000 0.306 0.347|0.327 0.371
96
0,386
0.405
10.386
0.114
0.119/0.113
0.179
0:16-
(0.384 0. 0.402 (0.36 0,100 0.651 0. 0,599 0.376 0.119 0.191 (0.199 0.159
192
0.441
0.436
00:45
0.424
0,446
0.44550.441
0.445 (1,471
0.474
0.525
0.442/0.456
0.492
(D.4%
0.429
0.432 0.719 0.631 0.420 0.448
0.534
0.504 00,500 D. 0.482
336
0.487
0.158
0.179
0,446
0.501
0.166 0.570
0.565 0.515
0.119 0.169 0.1 0.159 0.778
0.535 0.196 0.459 0.465 0.588
720
0.500
0.491
0.481
0.470
1.500
0.488
(16653
0.621 00.594 0.55x
0.55x 0.552 0.500 0.516 0.836 0.699 0.506 0.507 0.643 0.616 0,512
Avg
00.454
0.447
10
444
0.434
(0.
0.454/0.529 (0.329 0.522 io 545 0.507/o (o../oo..55 0. 0.450 0.345566044443 0.452 (. 0.747 0.647 0.440 4000 570 0.53770..440 0.487
0.454
96
0.299
0.349
10,288
0.338 0.3002
0.348
(1.745 0.584 0.400 0.440 34 0. 0.374/0.333 0.387 0.207 0.621 0.358 0.39770.4 0.436 0 0.458 0.388
192
0.380
0,400
0.374
0.390
0.385
0.400
0.877 0.656 0.528 0.509
0.402 0.414 (0.47 0.476 0.860 0.689 0 0:4 0.439 0.512
0.456 0.452
336
0.428
0.432
0.415
0.426 0.426 0.433 1 1.043
0.731 0.643 0.571 0.452 0.452 (0.544 0.541 1.000 0.744 0.496 0.487 0.552 0.551 0.482 0.486
720
0.427
0.445
0.420
0.440
0.431 0.446
1.104 0.763 0.874 0.679
0.462 0.468)0.831 0.657 1.249 0.838 0.463 0.474 0.562 0. 0.565 0.511
Avg
0.383
0.407
|0.374
0.395) 0.107 (0.912 0.681 0.611 0.550jo.114
0.127/0.555 0.515 0.951 0.723 0.437 0.49 0.526 0.516)0.150 0.159
96
0.148
0,240
10.201
0.281
181 0.270/.211 0,314 0.237 0.329/0. 1666 0.372/0.197 0.282 0.247 0.345 o. 193 0.30 0.
0 0.273/0.201 0.317
ECL
192
0.162
0.253
10:001
0.283
0. 188 0.274 (12311 0.322 0.2336
0.30000:184 0 0.289 10.19 0.285 0.257 0,355 o 0:315 0.1 0 0.286 0.222 0.334
336
0.178
0,269
0.215
0,298
0. 0.293 (1246 0.337 0.249 0,344 0.198 0.300 0,301 0.269 0.369 0.214 0.329 0.200 0 3.3 0.233 0,338
720
0.225
0.317
0.2257
0.331
[0 (1.446 0. 0.324/ (1.780 0.363 0.2284
0.375/0.230
0.375 (0.330 D. 0.320(0.245 0.333 0.299 O. 0.390 o 246 0.355 0.222 0.321 (0.2 0.361
Avg
00.178
0.270
10.219
0.298
10.005 0.290/0.244
0.334
0:251 0.344 jo.
0. 0.295/0.212 0.300 0.268 0.365 0.214 0.327/o oo 193 0. 0.296/0.227 0,338
Exchange
96
0.006
0.206
10.093
0.217 0.088
0.205
(0.256
0.367
0.094
0.218)0.1177
0.234/0.088 0.218
0.267 0.396
0.148
0.278
0.111 0.237/0.197 0.323
192
0.177
0.299
0.1
0.307
0.176
0.177
0.299
0.170
0,509
0.184 0.307
0.226 0.344 176 0.315 0,351 0.459
0.271
0.315
0.219 0.335 0,369
336
0.331
0.417
0.351
0.432
1,268
0.853
0.349
0.431
0.448 0.448 0.311 0.427
0.460
0.427
0.421
0.476 0.524
720
0.847
0.691
0.836
0.714 0.901
0.714
1.767
1.068 0.852 0.699 0.961 6.746,00.837
0.695
1.058 0.797
1.195 0.695
Lo92 0.769 L.11 0.911
Avg
0.3660
0.403
|0.378
0.417 10.367
0,404/0.940
(0,940
0,707 0.370 0.413/00.416 0. 0.443/0.354 0.414 0.750 0.626 0.519 0.429 0.461 0. 0.454 0,539
96
10.295
0.268
10 64699
389 (1.462 0.295 01522
0,200 io sus 0.403l0.3000 w0.90. 0.321 (0,650 0. 0. 787 0 4,49 o 0.587 o 366 6.612 o 0.333 0.388
Traffic
192
0.417
0.276
00.001
0.366 0,406 0.296 0.530 0.295 0.756
0.474 0.617
0.336 0.370 0.789
0 0.604 0.373 0.613 0 0.616 0.382
336
0.433
0.283
60600
369
[1483 0. 30. 0.558
0.305
0.762 0.477 0.622 D. 0..65 0.373 0.797 0.500 0.621 o 08383
3.630018 0.618 D 0.325 0.662 0.357
720
0.467
0.302
0.647
0.387
0.514
( 0,589
0.328
0.779 0.449 640 0.350 (.66 0.394 0.841 0.523 0.626 0.382 0.653 0.355 0.660 0.408
Avg
00.428
0.282
[0.626
0.378
08.48 0.304|0.550
0.304 0.760 0.473j0.620 0.336|0.625 0.383 0.804 0.509 0.610 0.62600624
0.340j0.628 0.379
Weather
96
0.114
0.214
10.192
1.77
0.218
10.158
0,230 0 0.202 0,261 10.17 0.220 0.255 0.221 0.306 0.217 0.296
2.966..173 0.173 0. 0.22360.266 0.336
192
0.221
0.254
0.220
0.271
0.225
0.259
00.206
0.277
0.242 0 0.299...111 1.311 0. 261 (0. 03.237 0.296 0.261 0,340 0 0.276 0.336 0.245 0 0.285 00307
0.28550.307 0.367
336
0.278
0.296
0.292
0.307
0.278
0.297
(0.272
0.335 0.287 0.335
0.150 0.306 00.3333333000 0.335 0,309 0,378 0.339 0.380 0.321 0.359 0.395
720
0.358
0.347
[0.
0.364 0,353
0.354 0.348
01.398
0.418 0.351 0.336 00366/0..666 0. 0.359 (0.345 0.381 00.77 0,427 0 403 0.428 0.414 0.4 0.410/0.419 0.428
Avg
|0.258
0.278
10.272 0.291
|0.259
0.281
(0.259
0.315
(02271 0.3200o
2559 0. 20.66 0.317 0.292 O. 0.363 O 309 0.360/o..20 288 0.3 0.314 0.382
Solar-Enee Energy
96
10.203
0.237
10:322
0.339
(.216
0.331
0:312
003990 250 0. 09200.290002222022
0.378 0.237 0.344 0 0.342/0.215
0.249)0.884 0.711
192
0,233
0.261
0.339
0.3560.267
0.310
0.725
0.339
0.416 0.296
0.318
0..10 020 0.398 0,280 0.380 0.285 0.380 0.255
0.272 0.692
336
0.248
0.273
0.3997 0.369
0.290
0.250
0.368
0.30
0.330 0.415
0.389 0.282 0.376 0.290 0.296
0.723
720
0.249
0.275
0.397
0.356 0.289
0.3566..89
0.317
0.769
0.765 0.370 0.425
0.338 0.335 0.337 (0.566 0.413 0.308 0.388 0.357
0.427 0.285 0. 0.295 0.29550.882 0.717
Avg
00.233
0.262
J0.369
0.367
0.611
0.639 0.347
0.417 0.319/0.330
0.401 0.282 0,375
0.381
[02661 0.38]
(0885
0.711
1" Count]
16
22
6
12
12
11
3
0
0
0
3
0
4
0
0
0
0
23

---

Published as a conference paper at ICLR 2024
Table 11: Full results of the Market dataset. We compare extensive competitive models on the real-world transaction forecasting task. Aug means the average results from all prediction lengths.
Models
Transformer
RLinear
PatchTST
Crossformer
TIDE
TimesNet
DLinear
SCINet
FEDforer
Stationary
Autoformer
(Ours)
2023
2023
F023
2023
2023
2023
20224
2022
20220
2021
Metric
MAE
MSE
MAE
MAE
MSE
MAE
MSE MAE
MAP
MSE MAE
MAE
MSE
MAE
12.
1.088
0.126
0.139
0.032
0.155 0.068 0.141
0.173
0.273 0.177 0.093 0.183
202 0.310
0.384 143 0.243 0.444
Merchant
24
0.138
0.079
0.091
0.170
0.274
0.195
0.200
0.323
0.378
0.269 0.636
72.
0.157
0.156
0.090
0.180
0.123
0.197
0.099
0.116
0.388
0.431
0.390
0.300 0.479
144
(.086
0.167
0.157
0.253
0:253
0.185
0.185
0.218
00.208
0.31
0.3110.091
0.091
0.183 0.18300.3 0.124
0.225
(0.459
0.477
p.359
0.453
0.294 0.566
Avg
|0.072
0.147
[0.152 0.247
0.084
0.171
0.117
0.181
18.187
0.289
0.003
0.184 0.110 0.2066.6...3311
0.385
0.299
0.401
172
0.277
0.494 0.531
12
10.189
0.205
0.419
0.411 0.255
0.250
0.270
(0.186
0.127
0.275
0.277
0.380
0.355
0.525
0.151
0.553
0.508
0.332 0.555
Wealth
0.255
0.244
0.543
0.446
0,320
0.291
0.329
0.233
0.545
0:463
0.285
0.456
0.397
0.588
0.5697
0.514
0.377
0.377
0.761 0.611
0.421
0.327
0.634
0.481
(1,459
0.484
0.324
0.665
0.510
0:526
0,555
01438
0.766
0.558
0.575
0.444 0.658
144
0.517
0.379
0.683
0.504
0.541
0.404
0.633
0.388
(0.698
0.526
0.383
0.611
0.459
0.568
0.744
0.604
0.637
0.498 0.627
Avg
|0.345
0.289
0.555
0.3991
0.326
0.129
0.288
(0.595
0.18:
|0.360
0.318
0.501
0.412
(0.660
0.514
(0.625
0.513
0.115 0.612
12
10.133
0.170
0.329
0.304
0.164
0.206
4.630
0.520
(0.512
0.350
0.291
0.321
0.271
1.865
0.602
1.537
0.535
0,384 1.651 0.593
Finance
24
0.155
0.197
0.386
0.332
0.
0.198
0.228 4.987
0.568
(0.635
0,383
0.503
0.2977
0.464
0,318 2.228
0.664
1,553
0.5470.551
0,586
1.671 0.594
72
0.212
0.240
0.446
0.353
0.268
0.273
5.631
0.675
1,239
0.544 0.310
0.986
0.423
3:084
0.793
1.612 0.554 2.004
0,853
2,054 0.758
144
0.257 1..445
0.429
0.355
0.293
0.286
6.083
0.708
1.562
0.5664
0.333
1.287
0.473
4.089
0.875
1.784
0.636
2. 2.379
0.947
2.114 0.778
Arg
10.184
0.216
(0.395
0.336
5.333
(0.987
0.442 0.308
0.765 0.372 0.734
0.734|
1.621
0.569
0.569|1.366
1.368
0.643]
1.872 0.681
12
10.051
0.127
0..668
0.272/0.068
0.164
0.055
0.140
(0.212
0.304/0.074
0,304
0.0074
0.169
0.096 0.198
19.199
0.301
|0.668
0,379l0, 1400 0.252/0.366 0.461
Terminal
24
0.059
0.139
0.185
0.274 0.173
0.065
0.155
0.2011
0,301 0.081
0.178 0.105 0.209 0.225
0.325
0.256
0,370 0.289 0,708 0.644
72
0.071
0.160
0.183
0.291
00081
0.187
0.077
0.120
(0.222
0.316
0.178 0.109 0.215 (0.117 0.338
0.285
o. 0.396
2020
0.321 0.510 0.552
144
(.079
0.171
10.184
0.292
0.193
0.085
0.181
(0.229
0.322
0.088
0. 0.113 0.2200.....0....... 0.425
0.372
0. 0.46x 0.20 0.322
(1.468 0,528
Avg
1..65
0.150
10.180
0.286 0.179 0.071
0.162
0.211
0.31 0.080
0.1 0.106 0.210/0.280 0.360
0.295
0.40s
0.100.110000000
0.296
0.518 0.547
12
10,050
0.121
1..13
0.230l0.065
0.156
0.152
0.145
10.184
0.265/0.094
0.265
00009
0.171.0.000
0.171 0.090
0.180/0.161
0.227
0.349l0.129
0.229|0.382 0.137
Payment
24
0.062
0.135
0.144
0.249
0.077
0.167 0.178
0.165
0.183 0.269
0.099
0.178 0.108 0.196 0.216
0.250
0.265
0.345
0.360.3345 33455..000 0.412
72
0.082
0.155
0.151
0:251
0.0044
0.188
01236
0.193
0.226
0.287
0.111 0.189 0.129 0.289 (1.
0.370
0.228
0.284
0.360
183
0.291
0.437 0.471
144
0.093
0.166
j.
1.1154
0.251
0.101
0.190
0.260
0.214
0.2340
0.5115 0. 0.29440.115
0.294
0.138 0.215 0.211500-4100000 (0-100 0.391
0.379
0.441
091
0.296
0.518
Avg
|0.022
0.144
14.143
0.245
0.084
0.174
0.207
0.179
(0.208
0.277
305 0.182 0.116 0.200/0.288 0.322
|0.00
0.377333333
0.373
0.271
00.477 0.460
12
10.065
0.129
10.191
0.247/0.091
(0.267 0.289
0.180
0.143
00.11000.10.000000 0.306
00.309
0.366l0.175
0.243
0.580
Customer
0.141
0.214
0.26440,107
0.107
0.173
0.293
0.2667
0.291
0.0
0.210 0.338 0.344
0.369
0.900.1888
0.26- 0.642
0.161
0.266
0.131
0.190 0.331
0..33
0.149
091
0.511
0.408
0.374
0.267
0.324 0.611
144
1.126
0.173
2222 0.268
0.141
0.195
0.368
0.226
(0.363 0.332
0) 0.2006
0.222
0.239
0.687
0.461
0.450
0.456
0.336 0.373
0.586
Avg
|0.094
0.150
(0.214
0.261|o.118
0.261
0.1188
0.180
0.309
0.194
(0.308
0.307|0.
142
0.191
0.184
0.219/04461
0.385
|..30
0.
|0.10|02220
20.242
0.301
0.669
0.593
1" Count
27
0
0
0
24

---

Published as a conference paper at ICLR 2024
G DISCUSSIONS AND FURTHER IMPROVEMENT
G.1 DISCUSSIONS ON ARCHITECTURE-FREE METHODS
Channel Independence (CT) (Nie al., 2023). regarding variates of time series independently and adopting the shared backbone, have gained increasing popularity in forecasting with performance promotions an architecture-free method. Recent works (Han al. 2023 Li al. 2023) found that while Channel Dependence (CD) benefits from higher capacity ideally, CI can greatly boost the performance because of sample scarcity, since most of the current forecasting benchmarks are not large enough. We think it is essential to make variates independent, especially when there are potential risks of embedding as mentioned in Appendix E.3 inducing the ideal model capacity of CD limited by the excessively localized receptive field. However, the essence of CI, regarding multivariate time series univariately, can lead to cine-con training and inference and become an obstacle to scalability.. Still, multivariate correlations not be explicitly utilized. Perpendicular to these works, iTransformer repurposes an architecture with the native Transformer modules to tackle the issues.
RevIN (Kim et al. (2021) and Stationarization (Liu et al.) 2022b) have been widely applied for the distribution shift -tattiittionaitt) as architecture- free techniques. These works strive to reveal the temporal dependency better. This is accomplished by layer normalization in iTransformer and still leaves further improvement for us to tackle the distribution shift.
G.2 DISCUSSIONS ON LINEAR FORECASTERS
Linear forecasters have natural advantages in modeling temporal dependencies. The dense weight- ing (Zeng et al. 2023 Li et al. (023) can reveal measurement-free relationships among the time points of the same variate. More advanced linear forecasters focus on structural point-wise model- ing (Oreshkin et al., 2019 Liu et al.) 2022a 2023). By contrast, i'Transformer is particularly good at forecasting high-ding high-dimensional time series (numerous variates with complicated correlations, which can be common and realistic for practitioners in real forecasting applications). For variate correlating, the embedding keeps the variate independent and the attention module can be applied to dig it out. Under univariate scenarios, iTrrasffrmer actually becomes stackable linear forecaster (attention degradation), which leaves further enhancement to exploit the temporal dependency better.
G.3 DISCUSSIONS ON TRANSFORMERS
We emphasize that ITransformer actually proposes a new perspective to think about the multivariate time series modality, specifically, how to consider the variates and the tokenization. We list several representatives in Figure [9] Transformer treats time series as the natural language but the time- aligned embedding may bring about risks in multi-dimensional series. The problem can be alleviated by expanding the receptive field. Although it is believed that Patching (Zhang & Yan! 2023 Nie et al. 2023) can be more fine-grained, it also brings higher computational complexity and the potential interaction noise between time-unalll unaligned patches. If the current embedding (implemented by MLP) enhanced with more inductive bias (such as TCN), it may handle more robust cases with the variate token paradigm and enjoy the flexibility of Transformer with changeable numbers of tokens.
We believe the capability and scalability of Transformer have stood the test by extensive fields, but there is still improvement room to elaborately design components based on the inverted architecture, such efficient attention for multivariate correlation, structural temporal dependency modeling under distribution shift, fine-grained variate tokenization and well-designed embedding mechanisms.
Value
Temporal Token
Variate Token
Patch Token
Token
Token
Token
Token
Token
Token
Token
Token
Token
Time
Transformer View
Transformer View
Crossformer/Pattee View
Figure 19: Tokenizations for multivariate time series modality of representative Transformers.
25


Reviews:
[{"review_id": "u92XrFEZgp", "reviewer": "Reviewer_hg6h", "summary": "This paper explored a new angle to apply Transformer model to the multivariate time-series forecasting problem. Without the modification of the original transformer component, the proposed iTransformer inverted the duties of the self-attention mechanism and the feed-forward network. In iTransformer, the feed-forward network was used for series encoding, while the self-attention mechanism captured the correlation among different variates. The authors conducted experiments on six real-world datasets to evaluate the proposed model.", "questions": "1.\tCan you please explain why the TiDE results are so different from those reported in their original paper?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tThis paper provided a simple and effective inverted view to improve transformer-based multivariate time-series forecasters.\n\n2.\tCompared with the previous use of Transformer structure (without invert), the iTransformer showed some advantages, including better generalization on unseen variates, and the desired performance improvement over enlarged historical information.  \n\n3.\tExtensive experiments on different multivariate time-series forecasting tasks were conducted for evaluation. The author compared the proposed model with various baselines, along with a comprehensive modal analysis.", "weaknesses": "1.\tAccording to Table 3 and Table 7, most of the result values are relatively small. This suggests that some marginal improvement may be susceptible to random factors (e.g., iTransformer v.s. PatchTST on ETT and Weather dataset, iTransformer v.s. SCINet on PEMS dataset). Therefore, I recommend reporting the standard deviation under different random runs and adding a significance test to provide further insights.\n\n\n2.\tAlthough the proposed efficient training strategy can reduce the required memory, it would still be better to compare its efficiency with linear models, since recent studies have indicated their advantages in both performance and efficiency.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FEsTt6kN2T", "reviewer": "Reviewer_yopU", "summary": "This paper proposes a simple variant of transformer for time series forecasting, where the embedding is applied on each time series and the attention is across each variate. The idea is simple and effective. The improved performance is shown on various real-world datasets. The paper is well-written and easy to read. The idea can be viewed as a principal way to be adopted on various transformer-based architectures. The numerical experiments on different architectures and analysis of representations/correlations greatly enhance the importance of this work.", "questions": "1. It would be better to describe the train-validation-test split in experiments, like training in past years and predicting in the next year, as it could be tricky for data pre-processing in time series forecasting and cause data leakage issues.\n\n2. After reading this paper, does the author implicitly assume the heterogeneity of variates is more important than temporal dependency in terms of forecasting? \n\n3. There are some interesting results in Table 3. Could the author comment on the not-so-good performance of the second row (both attention)?\n\n4. A minor one: in the left panel of Figure 6, there is a little jump on the red line from 336 to 720. Any reason why? Do multiple runs help?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The authors propose a principal way to apply the transformer-based model for time series forecasting. The idea is simple and effective, and the effectiveness is demonstrated via extensive experiments and ablation studies.", "weaknesses": "The paper is relatively short of explanation/justification about the effectiveness of such an approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vMNBdFMZl3", "reviewer": "Reviewer_HTy3", "summary": "In this paper, Authors propose to investigate why Transformer-based models do not seem to be as efficient as Linear-based models for Multivariate Time Series Forecasting (MTSF), while they are predominant in other AI domains. They suggest that the way Transformer is implemented for MTSF is inappropriate. To better benefit from Transformer architecture, they propose to tokenize dataset based on variate and not on timestep. It ends up modifying the input and the FFN. Their proposal also uses only an encoder compared to the vanilla Transformer architecture.\n\nAuthors conduct extensive experiments with several Linear-based and Transformer-based baselines to determine the performance of their proposal. These experiments are performed over 6 usual MTSF datasets along with 6 market-based datasets. Their proposal, iTransformer, appears to achieve best predictions for all datasets despite the selected prediction horizon.", "questions": "# Paper as is\nAs far as I am concern, the paper in its current version with some proof-reading and the following revisions can be Accepted.\n * In abstract “the duties of the attention mechanism” should be “the following duties of the […]” to make reading smoother and avoid readers from questioning on what duties means here.\n * The following claim “with potentially unaligned timestamps” need to be referenced or proven. Which datasets have unaligned timestamps, or by unaligned does Authors means that there are delays between values from variates? In other terms, are physical measurements unaligned (which is not the case in the considered dataset)? or do values that correspond to the same “event” appears at different timesteps for different variates (which means that there are delays, which might be the case for Traffic, but for the others, in my opinion, Authors would need to demonstrate it. For instance, in Weather (which are data from the same weather station), it is clearly not the case).\n * In my opinion, it is more appropriate to call Electricity dataset as ECL.\n * Reproducibility\n   * Avoid confusion and precise which ETT is used. It looks like it is ETTm according to appendix (15min frequency), but is it m1 or m2? Informer results looks like ETTh though…\n\n   * Also, precise which PEMS you use PEMS-Bay (occupancy ratio or speed in San Francisco Bay?) or other?\n * In my opinion, Figure 1 is focusing too much on iTransformer results. Why not having each axis of the web going from 1 (in the center) to 0? For better fairness in the visualization.\n * Figure 7 (b?) right part is difficult to understand without explanation of x and y axis, and what is the differences between Score Map and Correlations. What is important in this figure? How should reader interpret these lines. Why is it better than usual Transformer?\n * Figure 8 needs label of x axis.\n * “which can be attributed to the extremely fluctuating series of the dataset, and the patching mechanism of PatchTST may lose focus on specific locality to handle rapid fluctuation. By contrast, our proposed method aggregating the whole series variations for series representations can better cope with this situation” I would need proof for such a claim (even in appendix). First, showing this situation and second showing learning weight for iTransformer that shows it cope with the situation.\n\nProof-read is required for instance:\n * “this goal can be hardy achieved” I guess there is a typo here and Authors was aiming to write hardly?\n * “Soloar-Energy”- > Solar Energy\n\n\n# Toward a bigger impact\n\n## Additional baselines and larger scope\nNonetheless, I feel that Authors are missing the opportunity to have an even greater impact in the MTSF community (And having my contribution score going from Good to Excellent). Indeed, the proposal is very promising and is on a very important topic, i.e., how to consider variate in MTSF. Are they channel dependent (CD) or channel independent (CI), and if CI are projection perform commonly or individually? However, in my humble opinion, despite doing a good job to present their proposal and results with ablation study and visualization, Authors fail to really position their proposal in the landscape of the above question. It is true that they compare they work to PatchTST, and the CD/CI discussion, but RLinear or RMLP (depending on the dataset) [2] appears to also beat PatchTST. And especially, RLinear with individual projection (one linear layer per variate) similarly to NLinear or DLinear performs better. In addition, RLinear or RMLP use RevIN that was proposed in [1]. The latter show that RevIN helps to handle distribution shift and could be applied to Transformer-based models such as Informer to improve their performance.\n\nTherefore, in my opinion, the paper will have a greater impact if Authors compare their results to these following baselines: revInformer, RLinear and RMLP (and so corresponding papers). And discuss the results and impact of inverse versus CI versus distribution shift handling. This extra step would really be significant for the community in order to have a better understanding of the big picture and what is happening here.\n\nIn addition, the abstract and intro emphasis that Transformer superiority is “shaken” by Linear-based models. However, Authors have only few of such Linear models as baselines. Therefore, adding RLinear and RMLP, especially if iTransformer can beat them, will emphasis such a claim. \n\nAuthors cited [2] in their paper, so they are aware of this work and in my opinion should have included it.\n\nFinally, for the experiments where only P% of the variates are used for training, as the variates are selected randomly, it would be good to perform the experiment with different set of random variates and plot an error plot or box plot. This will further highlight that the set selected randomly is not a specific case. For instance, in Figure 8 (a? left one), we could have for each % of variate the min, max, average among the different sets. Figure 5 could be a boxplot. In addition, Authors should make clear that the set of variates use with iTransformer is the same set used in CI transformer (Figure 5) to avoid any misunderstanding from Reader.\n\n## More results in appendix to ensure proper reproducibility\nFurthermore, in order to target greater impact on the community, I would suggest Authors to add results and visualizations for the other datasets (similar to Table2, Figure 5, and following) in appendix. This would help to make sure results showed applied to all datasets. Indeed, Authors mentioned that PEMS is more difficult so the nature/type/characteristics of the dataset may impact the results and it would be important to mention it and discuss it (even though this point might what Authors expect to do as future work by saying “explore iTransformers for extensive time series analysis tasks”).\n\nEspecially, I also expect Figure 6 for dataset like Solar energy to not be as good as the others. Because seasonality of Solar energy is high and it strongly depend on the weather, so increasing the lookback window might not be that beneficial.\n\n\n\n[1] https://openreview.net/pdf?id=cGDAkQo1C0p\n\n[2] https://arxiv.org/pdf/2305.10721.pdf", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "In my opinion, this paper will have a high impact on MTSF research and community. They back-up their proposal with extensive experiments on several datasets and compared to multiple baselines. \nThe results and ablation study are discussed even though I feel it could have gone further, but the page limitation was probably the issue.", "weaknesses": "The biggest weaknesses in my opinion are first, the writing style, which sometimes is not going to the point (could be improved to better convey the idea and have greater impact) and second that Authors fail to include some important SOTA baselines and properly synthetize/discuss CD/CI/distribution shift comparing all the baselines (including the missing ones) and iTransformer.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bRklD21RqZ", "reviewer": "Reviewer_b5RW", "summary": "The paper addresses the problem of time series forecasting.\nThe authors propose a model that embeds univariate\nchannels as a whole and then uses attention between\nembedded channels. In experiments on several datasets\nthey show that their model outperforms current models\nand established new state of the art results.", "questions": "The paper proposes a very simple idea, but the experiments\nto the best of my knowledge are establishing a new state of the\nart, making it an important contribution like an indepth study.\nI also liked the ablation study with growing observation horizons,\nas they now are more plausible than in the related work: longer\nobservation horizons usually pay off for your model.\n\nSome points should be discussed:\nw1. experiments only on selected datasets compared to some\n  major baselines.\n- You do not report on datasets Exchange, ILI and ETTm1, ETTm2,\n  ETTh1 and ETTh2, different from the experiments reported in TimesNet.\n  This way it is hard to see if the proposed method really outperforms\n  the baselines consistently or just on the selected datasets.\n\nw2. experimental results for close baseline PatchTST varies from\n  published results.\n- PatchTST consistently reports better results, e.g., for Electricity\n  with horizon 96 they report an MSE of 0.129, you report 0.195.\n  Where does the difference come from? \n\nw3. no standard deviations reported.\n- Standard deviations will help to assess which differences might be\n  significant and which spurious. \n\nSome minor language issues:\n- abstract, \"However, Transformer is challenged\": missing \"a\".\n- p. 2 \"irrationality\": sounds a little bit too strong to me.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "s1. very simple idea and model.\ns2. very good results, establishing new state of the art results.\ns3. interesting additional finding that longer lookback windows\n  now are mostly beneficial (fig. 6).", "weaknesses": "w1. experiments only on selected datasets compared to some\n  major baselines.\nw2. experimental results for close baseline PatchTST varies from\n  published results.\nw3. no standard deviations reported.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)