UniTST: Effectively Modeling Inter-Series and Intra-Series Dependencies for Multivariate Time Series Forecasting
Authors: Juncheng Liu
Chenghao Liu
Gerald Woo
Yiwei Wang
Bryan Hooi
Caiming Xiong
Doyen Sahoo
Venue: Accepted by TMLR ()

Abstract: Transformer-based models have emerged as powerful tools for multivariate time series forecasting (MTSF). However, existing Transformer models often fall short of capturing both intricate dependencies across variate and temporal dimensions in MTS data. Some recent models are proposed to separately capture variate and temporal dependencies through either two sequential or parallel attention mechanisms. However, these methods cannot directly and explicitly learn the intricate inter-series and intra-series dependencies. In this work, we first demonstrate that these dependencies are very important as they usually exist in real-world data. To directly model these dependencies, we propose a transformer-based model UniTST containing a unified attention mechanism on the flattened patch tokens. Additionally, we add a dispatcher module which reduces the complexity and makes the model feasible for a potentially large number of variates. Although our proposed model employs a simple architecture, it offers compelling performance as shown in our extensive experiments on several datasets for time series forecasting.

Published in Transactions on Machine Learning Research (06/2025)
UniTST: Effectively Modeling Inter-Series and Intra-Series Dependencies for Multivariate Time Series Forecasting
Juncheng Lin'
Chenghao Lin'
Gerald Woo Doyen Sahoo
Yiwei Wang²
Bryan Hoof
Caiming Xiong
juncheng. larog....uuur ww..u..u..ee..
Salesforce'
University of California, Merced 2
National University of Singapore
Reviewed on OpenReview: http:://beeerreddwwwwtt///ooooot
Abstract
Transformer-based models have emerged as powerful tools for multivariate time series forecasting (MTSF). However, existing Transformer models often fall short of capturing both intricate dependencies across variate and temporal dimensions in MTS data. Some recent models are proposed to separately capture variate and temporal dependencies through either two sequential or parallel attention mechanisms. However, these methods cannot directly and explicitly learn the intricale inter-series and intra-series dependencies. In this work, we first demonstrate that these dependencies are very important as they usually exist in real-world data. To directly model these dependencics, we propose a transformer-basee model UniTST containing a unilied altention mechanism on the flattened patch lokens. Additionally, we add dispatcher module which reduces the complexity and makes the model feasible for potentially large number of variates. Although our proposed model employs a simple architecture, it offers compelling performance shown in extensive experiments several datasets for time series forecasting.
1 Introduction
Inspired by the success of Transformer-based models in various fields such as natural language processing
Touvroo et al. 2023a Chiang et al.] 2023
Almazrouei et al. 2023 MosaicML 2023 Touvron et al. 2023b
OpenAl 2022 Google
2023
Touvron et al.
2023b) and computer vision (Wu et al.) 2020 Liu et al.
2021b
Jamil et al. 2023),
Transformers have also
garnered much attention in the community of mullivariate lime
series forecasting (MTSF) (Nie et al. 2023
Liu et al. 2024 Wu et al.) [022]] Zhang & Yan
2023
Zhou et al,
2022
Carlini al.
2023 Han et al.
2024
Pioncering works (Li et al.) 2021 Wu et al.
2021
Zhou al.
2022
treat multiple variates (aka channels) at each time step the input unit for transformers, similar to
tokens in the language domain, but its performance was even inferior to linear models (Zeng et al. 2023
Han et al. 2023). Considering the noisy information from individual time points, Veriaate-Indepeneen and Patch-Base Based (Nic et al. 2023) methods are subsequently proposed and achieve positive results by avoiding mixing noises from multiple variates and aggregating information from several adjacent time points as input. Nevertheless, these mothods neglect the cross-varian variate relationships and interfere with the learning of temporal dynamics across variates.
To tackle this problem, iTTranffrmerr Liu et al. 2024 embeds the entire time series of a variate into token and employs "variate-wise attention" to model variate dependencies. However, it lacks the capability to model intra-variate temporal dependencies. Concurrently, several approaches (Zhang & Yan 2023 Carlini et al. 2023 Yu et al. 2024) utilize both variate-wise attention and (aanhhpchh)wwize attention to capture inter-variate and intra-variate dependencies, either soquentially or parallelly. Yet, they may raise the difficulty of modeling the diverse time and variate dependencies the errors from one stage can affect the other stage and ealntually the overall performance.

---

Published in Transactions on Machine Learning Research (06/2025)
Time-wise attention over time each individual variate
attention over variates the time
Input multivariate time series
Patching
Time-wise Altn
@Embedding
Previous models apply two attentions either sequentially parallelly
Time-wise
Variate-
Output
Output
Time
wise Attn
Attention cross-time cross-variate all once
Our model with unified attention
Variates
Patch token tensor
Figure 1: Comparison between our model and previous models. Previous models apply time-wis attention and variate-wise attention modules either sequentially or parallelly, which cannot capture cross-time cross-variate dependencies (i.e., green links) simultaneously like our model.
Additionally, either two parallel or sequential attention mechanisms cannot explicitly model the direct dependencies across different variates and different times, which we show in Fig- ure [ Regardless of how previous works apply time-wise attention and variate-wise atten- tion parallelly or sequentially, they would still lack the green links to capture cross-time cross- variate dependencies (aka inter-series intra-series dependencies) To further explain, as we illustrate in Figure 2] the time series of variate during period shares the same trend with the time series of variate 2 during period 2. This type of correlations cannot be directly modeled by previous works as it requires directly modeling tross-time pross-variate dependencies simul- tancously. This type of correlation important as it generally exists in real-world data as we further demonstrate in Sec X
simultaneously as in our model.
Variata
Values
To mitigate the limitations of previous works, in this paper, revisit the structure of multivariate time series transformers and propose lime series transformer with unified altention (URiTST) as fundamental backbone for multivariate forecast- ing. Technically, we flatten all patches from different variates into a unified sequence and adopt the attention for inter-variate mitigate the
Veriste
Redod
Panod
Pirtod
Patod4
Figure 2: Explicit correlation between two sub-series at different periods from two dif- ferent variates (i.e., strong correlation be- tween period of variate 1 and period and variate 2).
and intra-variate dependencies simultancously. high memory cost associated with the flattening strategy, further develop dispatcher mechanism to reduce complexity from quadratic to linear. Our contributions summarized as follows:
We point out the limitation of previous transformer models for multivariate time scries forecasting: their lack of ability simultancously capture both inter-variate and intra-variate dependencies. With evidence in world data, demonstrate that these dependencies are important and commonly exist.
2

---

Published in Transactions on Machine Learning Research (06/2025)
To mitigate the limitation, we propose UniTST as simple, general yet effective transformer for modeling multivariate time series data, which flattens all patches from different variates into unified sequence to effectively capture inter-variate and intra-variate dependencies.
Despite the simple designs used in UniTST, we empirically demonstrate that UniTST achieves state- of-the-art performance on real-world benchmarks for both long-term and short-term forecasting with improvements up to 13%. In addition, we provide results of the ablation study and visualization to further demonstrate the effectiveness of our model.
2
Related Work
Recently, many Transformer-baseed models have been also proposed for multivariate time series forecasting and demonstrated great potential (Liu et al.
2021a Wu et al. 2021 Li et al.
2021
Zhang & Yan 2023 Zhou et al.
2022 Li et al.
2019 Luo & Wang 2024
Zhang et al. 2024 Ilbert et al.
2024). Several approaches (Wu et al.
2021
Li et al.
2021
Zhou et al. 2022
embed temporal tokens that contain the multivariate representation
of each time step and utilize attention mechanisms to model temporal dependencies. However, due to the
vulnerability the distribution shift, these models with such channel mixing structure are often outperformed by simple linear models (Zeng et al. 2023 Han et al.) 2023). Subsequently, PatchTST Nie et al. 2023) considers channel independence and models temporal dependencies within each channel to make predictions independently. Nonctheless, it ignores the correlation between variates, which may hinder its performance.
To model variate dependencies, in the past two years, several works have been proposed (Liu el al.) 2024
Zhang & Yan 2023 Carlini et al. 2023 Han et al., 2024 Yu et al., 2024, Wu et al. 2023). iTransformer (Liu et al.] 2024) models channel dependencies by embodding the whole time scries of variate into a token and using "variate-wise attention" without explicitly modeling on temporal dependencies. ElasTST (Zhang et al., 2024) tackles varied-horizon forecasting with non-autoregressive design and horizon-invariant embeddings
while DeformableTST (Luo & Wang 2024h reduces Transformers' reliance on patching by using deformable attention.
Additionally, several methods proposed different modules to capture both time and variate dependencies. However, they can either sequentially or parallelly capture time and variate dependencies and are not able to capture them simultancously. In the later Section B we show the importance of simultancously capturing both time and variate dependencies by providing empirical evidence in real-world data. Crossformer (Zhang & Yan 2023 uses the encoder-decoder architecture with two-stage attention layers to sequentially model cross- time dependencies and then cross-variate dependencies. CARD (Carlini et al. 2023) employs the encoder-only architecture utilizing a similar sequential two-stage attention mechanism for cross-time, cross- channel dependencies and a token blend module to capture multi-scale information. Leddam (Yu et al.) 2024) designs a lcarnable decomposition and a dual attention module that parallelly model inter-variate dependencies with "channel-wise atlention" and intra-variate temporal dependencies with "aulo-regressive attention". In summary, these works generally model (either sequentially or parallelly), and aggregate these two types of information to get the outputs. In contrast, our model has a general ability to directly capture inter-variate and intra-variate dependencies simultaneously, which is more effective. We provide more discussion on the comparison in Section 4.2
and inter-variate dependencies separately
Moreover, CATS (Lu et al. 2024h constructs auxiliary series and capture inter-series dependencies from auxiliary series. In contrast, our method is applied directly on the original series with considering all multivariate as a unified sequence. CrossGNN (Huang al. 2023), as a GNN-based method, proposes a cross interation layer to capture cross-scale interation on the time dimension and cross-variate interaction on the variate dimension. However, it still relies on sequential manner to capture cross-time and crois-variate dependencies. Similar to it, TimeXer (Wang et al.) 2024) sequentially capture gross-time and cross-variate dependencies by ingesting external information from exogenous variables. With the same goal as our work, LIFT Zhhao & Shen 2024) also aims to capture cross-time and cross-variate dependencies simultaneously. However, it requires directly calculations on leading indicators for each pair of variates and applies leading indicators lagged variates, which may need massive computational costs. In contrast, our proposod method UNiTST can model the cross-time and cross-variate from the time series sequence without explicit calculation leading indicators.
3

---

Published in Transactions on Machine Learning Research (06/2025)
3 Preliminary and Motivation
In multivariate time series forecasting, given historical observations X00+ E RRNXL with L time steps and N variates, the task is to predict the future S time steps, i.c., +. E xxx, For convenience, we denote X, = ( as the whole time series of the i-th variate and x as the recorded time points of all variates at time step t.
To illustrate the diverse cross-time and cross-variate dependencies from real-world data, we use the following 2.2422 to mcasure it,
correlation coofficient between (() and x
Definition 1 (Cross-Timee Time Cross-Variate Variate Correlation Coefficient)..
RO,D), X x(1)
55)
1
1+b 2
X44h nنن
(1)
T.
rb1
where 1 and of) are the mean and standard deviation of corresponding time series patches.
Utilizing the above corrclation cocfficient,
quantify and fur-
Saten Sater-nneryy Energy vars (0.10)
ther understand the diverse cross-line cross-variale correlation. We
visualize the correlation coefficient between different time periods from two different variates in Figure 3 We split the time series into
0.8
several patches and each patch denotes a time period containing 16 time steps. In Figure 3 we can see that, first, given a pair of variates, the inter-variate dependencies are quite different for different patches. Looking at the column of Patch 20 in variate 10, it is strongly corre- lated with patch 3, 5, 11, 20, 24 of variate 0, while it is very weakly correlated with all other patches from variate O. It suggests that there is no consistent correlation pattern for different patch pairs of two variates (i.e., not all the same coefficient at a row row/colum /column the correlation map) and inter-variate dependencies are actually at the fine-grained patch level. Therefore, previous transformer- based models have deficiency in directly capluring this kind of dependen- cies. The reason is that they either only capture the dependencies for the whole time series between two variates without considering the fine-grat grained temporal dependencies across different variates (Lin
15
02
00
Figure 3: Correlation between patches from different variates. x-axis: patch indices in variate 10, y-axis: patch in- dices in variate 0.
Yu et al., 2024] which are indirect and unable to explicitly learn these dependencies. In Appendix A we provide more examples to demonstrate the ubiquity and the diversity of these cross-time cross-variate correlations.
et al. 2024) or use two separate attention mechanisms (Zhang & Yan 2023:
Carlini et al. 2023
Motivated by the deficiency of previous models in capturing these important dependencies, in this work, aim to propose model with the ability to explicitly directly capture cross-time cross-variate interactions for multivariate data.
4 Methodology
In this section, we describe our proposed Transformer-based method (UniTSTT for modeling inter-variate and intra-variate dependencies for multivariate time series forecasting. Then, we discuss and compare model with previous Transformer-based models in detail.
4.1 Model Structure Overview
We illustrate our proposed UniTST with a unified attention mechanism in Figure
Embedding the patches from different variates the tokens Given the time series with N variates X NX divide each univariate time series xt into patches as in Nie et al. (2023) Zhang & Yan (202)).
4

---

Published in Transactions on Machine Learning Research (06/2025)
Multivariate Output
Self-Attention without dispetchers
Norm
Projection
(c)
Feed fera-forward
Transformer Encoder
Norm
Attention with dispatchers
Patching
Flatten Patches
(d)
Unified
Embedding
Attention
Enseeiinn Dimeriion
ponches per warate
(a)
(b)
Figure Framework Overview. We flatten the patches from all variates into a sequence as the input of the Transformer Encoder and replace the original self-attention with the proposed unified attention with dispatchers to reduce the memory complexity.
With the patch length I and the stride 8, for each variate i, we obtain patch sequence 2/ Rex! where p is the number of patches. Considering all variales, the lensor containing all patches is denoted as X, RNxxxl where N the number of variates. With each patch as token, the 2D token embeddings are generated using a lincar projection with position embeddings:
Embedding((X) Xp)) = X Wpos E Nxpxd
(2)
where W E Rii is the learnable projection matrix and Wpos E Nxxxx is the learnable position embeddings. With 2D token embeddings, we denote (i.k) the token embedding of the k-th patches in the i-th variate, resulting in N x p tokens.
Self attention on the flattened patch sequence Considering any tokens, there two relationships: 1) they are from the samne variate; 2) they are from two different variates. These represent intra-variate and cross-variate dependencies, respectively. A desired model should have the ability to capture both types of dependencies, especially cross-variate dependencies. To capture both intra-variate and cross-variate dependencics among tokens, we flatton the 2D token embedding matrix H into a 1D sequence with N x p tokens. We use this 1D sequence X' E RRN xd as the input and feed it to a vanilla Transformer encoder. The multi-head self-attention (MSA) mechanism is directly applied to the 1D sequence:
QKT Vde
0 MSA(Q, K, V) Softmax
)V,
(3)
with the query matrix Q= = X'Wo E (NXx)xdd the key matrix K = X'Wx E (R(xx)xxd, the value matrix V = X'Wy E xd and Wo, WK E dddde Wy E [dxd The MSA helps the model to capture dependencies among all tokens, including both intra-variate and cross- cross-variate variate dependencies. However, the MSA results in an attention map with the memory complexity O(N2p?), which is very costly when we have a large number of variates N.
Dispatchers In order to mitigate the complexity of possible large N, we further propose dispatcher mechanism to aggregate and dispatch the dependencies among tokens. We add k(k << N) learnable embeddings dispatchers and use cross attention to distribute the dependencies. The dispatchers aggregate the information from all tokens by using the dispatcher embeddings D as the query and the token embeddings as the key and value:
DW (X'WK, IT )
D' = Attention(DWo, X',,,, = Softmax
(4)
Vdn
5

---

Published in Transactions on Machine Learning Research (06/2025)
where the complexity is OkkNp). After that, the dispatchers distribute the dependencies information to all tokens by setting the token embeddings as the key and the dispatcher embeddings as the key and value:
X'W X'Wa, )D'W
O' = AttentionnX D'W,,, K2, D'Wv,) Softmax
(5)
dd
where the complexity is also O(kNp). Therefore, the overall complexity of our dispatcher mechanism is O(kNp), instead of O(N2p²) if we directly use sclf-attention the flattened patch sequence. With the dispatcher mechanism, the dependencies between any two patches can be explicitly modeled through attention, no matter if they are from the same variate or different variates.
In a transformer block, the output of attention O' is passed to a BatchNorm Layer and a feedforward layer with residual connections. After stacking several layers, the token representations are generated as ZNxD Tn the end, a linear projection is used to generate the prediction X E Rxs..
Loss function The Mean-Squared Error (MSE) loss is used as the objective function to measure the difference between the ground truth and the generated prodictions: L= Ns Σ N (X -X,44L1444)
4.2 Discussion and Comparison with Previous Models
Our proposed model is an encoder-only transformer model containing a unified attention mechanism with dispatchers. The model explicitly learns both intra-variate and inter-variate temporal dependencies among different patch tokens through attention, which means that it can directly capture the correlation between two time series at diffcrent periods from different variates. In contrast, these dependencics cannot be dircctly and explicitly captured by previous works which claim that they model variate dependencies (Liu et al. 2024
Zhang & Yan 2023 Carlini et al.) 2023 Yu al. 2024).
For example, iTransformer (Liu et al. variale as a token. It loses the ability to capl ure the fine-grained temporal dependencies across channels or within a channel. Crossformer (Zhang & Yan 2023) and CARD (Carlini et al., 2023) both propose to use a soquential two-stage attention mochanism to first capture dependencies on time dimensions and then capture dependencies on variate dimensions. This sequential manner does not directly capture cross-time cross-variate dependencies simultaneously, which makes them less effective as shown their empirical performance. In contrast, our proposed model uses more unified attention on flattened patch sequence with all patches from different channels, allowing direct and explicit modeling cross-time cross-variate dependencics. In addition, Yu et al. (2024) propose dual attention module with an i'Transformer-ikke encoder to varereve dependencies and an auto-regressive solf-attention on each channel to capture intra-variate dependencies separately. In this way, it also cannot directly capture cross-variate temporal dependencies between two patch tokens at different time steps from different variates (e.g., (i.)), while our model is able to directly capture these dependencics.
2024 captures variate dependencies using the whole time scries of
Worth noting that our proposed model is more general case directly capture intra-variate and inter-variate dependencies at more fine-graing grained level (i.e., patch level from different variates at different times). Moreover, our model employs simple architectures that can be casily implemented while the empirical results show the effectiveness of our model in Section 5.1 Additionally, we provide the analysis of computational complexity in Appendix B
5 Experiments
We conduct comprchensive experiments evaluate our proposed model UniTST and compare with representative basclines for both short-term and long-term time series forecasting on 13 datasets. Additionally, further dive deeper into model analysis to examine the effectiveness of our model from different aspects.
6

---

Published in Transactions on Machine Learning Research (06/2025)
5.1 Forecasting Results
We conduct extensive experiments to compare our model with several representative time series models for both short-term and long-tern ter-termmne time series forecasting.
Experimental Setting We conduct all the experiments with PyTorch (Paszke et al. 2019 and utilize a single NVIDIA A100 GPU with 40GB memory. We describe the hyperparameter choices used in our experiments in the following. For the optimizer, we use ADAM (Kingma & Ba 2015) with the learning rate in {10-3, 5 x 10-4, 1)= The batch sizes are selected from {16, 32, 64, 128} depending the dataset sizes. The maximum number of training epochs is set to 100 as in Nie et al. (2023). Meanwhile, we also use the early stop strategy to stop the training when the loss does not decrease in 10 epochs. The number of layers of our Transformer blocks is selected from (2,3,4] The hidden dimension of D is set from {128, 256, 512].
For the experimental results of our model, we report the averaged results with 5 runs with different seeds. For the results of previous models, we reuse the results from iTrransfrmer paper (Liu et al. 2024) as we are using the same experimental setting.
Baselines We sclect 11 well-known forccasting models as our basclines, including (1) Transformer-base
models: iTransformer (Liu et al.
2024), Crossformer ZZhang & Yan 2023). FEDformer (Zhou et al. 2022
Stationary (Liu et al., 2022b). PStchTST (Nie et al. 2023); ;; (2) Linear-based methods: DLinear (Zeng et al.
2023), RLinear (Li et al.) 2023
TiDE (Das et al. ) (3) Temporal Convolutional Network (TCN)-bases
mothods: TimesNet (Wu al.
2023), SCINet (Liu et al. 2022a ; and (4) MLP-based method: SOFTS
Han
et al. 2024).
Table 1: Multivariate long-term forecasting results with prediction lengths S E {96, 192, 336, 720} and fixed lookback length T = 96. Results are averaged from all prediction lengths. Full results are listed in
Appendix C.3.1 Table G
Models
UnITST
SOPTS
Flbaradormet
RLincwr
PStchTST
Croseformer
TIDE
TimesNot
DLinear
SCTNet
FEDformer
Stationary
(Oues)
[2024
2024
2023
2123
2026
2023
2022
X123
2022a
2122
202216
Metric
MSB
MAE]
MSE
MAB
MSE
MAE
MSK
MAE
MSE
MAP
MSE
MAE|
MSE
MAE MSE
MAE]
MSE
MAE
MSE
MAP
MSB
MAR
MSE
MAB
BCL
0.186
0.262
0.174
0.264
(0.178
0.270
10:219
0.208
jo
0.200
20.244
0.334
(02251
0.192
0.296/0.212 0.212
0.206
0.300 0.268 0.365
0.214
0.527
0.193
0.296
ETTml
0.379
0.394
0.393
0.403
[0.407
0.410
|0.414
0.407
10.387
0.400
[0.518
0.496
[0.419
0.419
0,400
0.406Jo.003
0.406
(o.403
0.407 0.486 0.481
[0.488
0.452]
0.481 0.456
ETTm2
0,280
0.326
0.267
0.330
[0.288
0.552
[0:288
0.327
|
0.326
|0775
0.610
|0,358 0.604
0.291
0.3333
0.333/00.350
(0.350 0.401 0.571
0.537|
00.505
0.349
0.306
0.347
ETTh1
0.442
0.435
0.449
0.442
[0.154
0.447
|0:466
0.434
0.454
|0.529
0.502
0..541 0.507 0-458
0.150
(0.458
0:452 0,747 0.647
|0.440
0.460)
0.570
0.537
ETTh2
0,363
0,390
p.a7a
0,400
[n,383
0.507
|0.374
0.299
|0,387
0.407
J0.922
0.084|
[0.011 0,550 0.414
0.427 |0,339 0.515 0.984 0.723) 00,437
0.449) 0.520 0.516
Exchange 0.351
0.398|
|0.360
0. 00000
10.378
0.017
/0.367
0.404
90.940
0,707 |0.370 0.413 0.416
/030//.35 0.434 0,750 0.626 0.5119 0. 0.129 0.161 0.154
Traflic
0.439
0.274
(0.400
0.266/0.122
0.267/
(0.128
0.282
|0.62G
0.378
|0.481
0.304
|0.500
0.304 (0.700 0.473 0.020
0.300
0.300/o.525 (0.52 0.383 0.804 0.509 0.610 0.376 0.004 0.340
Weather
0.242
0.271|
1,255
0.278
(0.258
0.208
(0.272
0.291
|0.259
0.281
20.259
0.315|0.271
0.315
(0.271 0.520 0,259 0.287/0.265 (0.265 0.317 0,252
0,363| |0,30 0.360
0.288
0.514
Solar. Energy 0.225
0.280
[0.229
0.256/0.233
o 0.202
(0,300
0.356
10:270
0.307
jo.611
0.630
(0.347
0.417
0:301 0.319/0.330 0.401 0.282 0.375 (0.291
0.381
(0.261 0.381
1st Count
Long-term forecasting Following I'Transformer (Liu et al. 2024 we use 4 different prediction lengths (i.e., (96, 192, 336, 720}) and fix the lookback window length as 96 for the long-term forecasting task. We evaluate models with MSE (Mean Squared Error) and MAE (Mean Absolute Error) - the lower values indicate better prediction performance. We summarize the long-term forecasting results Table | with the best in red and the second underlined. Overall, we can see that UniTST achieves the best results compared with 11 baselines on 7 out of 9 datasets for MSE and 8 out of 9 datasets for MAE. Particularly, iTransformer, state-of-the-a
the previous model, performs worse than our model in most cases of ETT datasets and ECL dataset (which are both from electricity domain). This may indicate that only model multivariate correlation without considering temporal correlation is not effective for some datasets. Meanwhile, the results of PatchTST are also deficient, suggesting that only capturing temporal relationships within a channel is not sufficient as well. In contrast, our proposed model UniTST can better capture temporal relationships both within a variate and across different variates, which leads to better prediction performance. Besides, although Crossformer is claimed to capture troes-time and variate dependencies, it still performs much worse compared with our
7

---

Published in Transactions on Machine Learning Research (06/2025)
approach. The reason is that their sequential design with two attention modules cannot simultaneously and effectively capture cross-time and cross-variate dependencies, while our approach can explicitly model these dependencies at the same time.
Table 2: Full results of the PEMS forecasting task. We compare extensive competitive models under different prediction lengths following the setting of SCINet (2022a). The input length is set to 96 for all baselines. Ang means the average results from all four prediction lengths.
Models
UNITST
SOFTS
iTransfommrr
RLincar
FatchTST Croseformer
TimesNet
DLincar
SCINet
FEDformer
Stationary
(Ours)
2024
2024
2023
2023
2023
2021
2023
2020
2022n
2022
20/2h
Metric
MSB
MAK
MSH
MAR
MSK
MAE
MSE
MSK
MAR
MAb
MSE
MAK
MSK
MA
MSH
MAR
MSB
MAR:
MSK
MA
MSK MAB
PEMS07 PEMS04 PEMS03
12
0.059
0.160
0.081
0.165
0.071
0.174
00. 12.126 0.236 0.099
0.216
0.090
0.203
0..178
0.305 0.085 0.192
0.122 0.243
0.060
0.172
0.126
0.251
0.088 0.188
24
0.074
0.180
0.084
0.18%
0.093
0.2001
0.246 0.334 0.142
0.259001121
0.240
0:257
0.371 0.1188 0.223
0:201 0.317
0.085
0.196
0.149
0.275
00.106 0:214
48
0.101
0.213
0.111
0.253
0.125
0.296
00.551 0.529 0.211 0.319
0.212
0.317
00.379
0.463 0.155 0.260 0.333
0.125
0.127
0.238
0:227
0.348
0.154 0:257
0.151
0.261
0:164
0.275
1.057 0.787 0.263 0.370
0.370/0:222
0.367
00.400
0.533 (0.228 0:317 0:457
0.515
0.178
0.287
0.348
0.434
00:247 0.336
Avg
0.097
0.204
0.104
0.210
0.113
0.221
[0.495
0.472 0.180
0.291 00.169
0.281
[0.326
0.419
0.419/0.147 0.147 0.248
(0.778
0.375
0.114
0.224
[0.213
0.327
(0.147 0.249
12
0.070
0.172
0.074
0.176
0.007
0.183
138
0.2252 0.105
0.224
10.008
0.218
(0:219
0.340 0.087 0.196
0.148 0.2772
0.073
0:177
00.138
0.262
(0.088 0.196
0.082
0.189
0.096
00:258
0.153
0.256
00:292
0.398 0:215
0.340
0.081
0.108
0:177
0.293
0.100 0:216
48
0.104
0.216
0.110
0.219
0.120
0.233
00.572
0.544 0.229
0:205
0.326
00.409
0..778 0.136 0.250
0.355
0.437
0.099
0.21
0.270
0.368
00.197 0:251
96
0:137
0.256
0.135
0.244
0.150
0.2802
1.
1.1 0.820 0.291
0.380
0.457
(0.4402 0.6322 00.180 0.303
(1.452
0.504
0.11144 0.227
00:341 0.427
(0.186 0:207
Arg
0.098
0.208
0,102
0.208
0.111
0.221
[0.526 0.491 0.195 0.307 20220
0.314
[0.353
0.437 (0.199 0.241 [0.295 0.385
0.092
0.202/0.220
0.2311 0.387 (0.127 0.200
12
0.057
0.158
0.057
0.152
0.165
118 0.235 0.095 0.207 0.004
0.200
(0,173
0.304 0.082 0.181 0.115 0.242
0.068
0.177
10.109
0.225
000083 0.186
21
0.075
0.174
0.073
0.175
0.190
00.202 0.311 0.150 0,262
0.192
0.139
0.217
(0,271
0.383 0.101 0.201 (0:210 0.329
0.119
0.225
0.125
0.244 00.102 0:207
0.107
0.208
0.096
0.110
0.215
00:562
0.541 0.253
0.341
0.369
(0.446
0.495 0.134 0:228 0..30 0.458
0.149
0.237
0.165
0.288
0.136 0.240
96
0.133
0.228
0.120
0.218
0.245
1.096
0.795 0.346
0.404 0.226
0.442
(0,629
0.577 0.181 0.279
0.394
0.553
0.141
0.204
0.262
0.376
0.187 0:287
Avg
0.093
0.191
000087
0.184
0.204
[0.501 0.478 0.211 0.308|0.235
0.315
|0.380
0.440 0.124 0.225 (0.329
0.440
0.395
0.119
0.294
00.105
0.283
0.112 0.230
PEMSIS
12
0.073
0.174
0.004
0.173|0.079
0.182
10.133 133
0:247 0.168 0.23120.116
0.214
[0.227
0.343/0.112 0:212 0.115 01.776)
0:212/0:154
0.087
0.184
(0.178
0.277
(0.109 0:207
0.096
0.197
0.104
0:201
0.115
0.219
00:249 0.343 0.234 0.281
0:211
0.260
00.118
0.409 (.144 0.238 0.248
0.353
0.122
0.221
0.211
0.301
00.140 0:236
48
0.141
0.239
0.164
0.253
0.186
0.235
00.069 0.544 0.3321 0.354
00.315
0.335
00.4077
0.570 0.1588 0:283 01.400 0..770
0.189
0.270
0.320
0.304
(0.211 0:2014
96
0.210
0.275
0.211
0.253
0.2530.221
0.221
0.267
1.
1.166 0.814 0.408
0.417
0.377
0.397
(0.721 0.502 00.320 0.351 (0.674 0.563
0.236
0.300
0.442 0.465
(0.345 0.367
Arr
0.130
0.221
1.88
0.219/0.150
0.226
[0.529
0.487
0,280
0.321
|0.208
0.307
[n.441
0.464
0.198 0.271
|0337
0.416
0.158
0.244
|0.286
0.358
(0.201 0.276
1 Count
11
0
0
0
0
0
0
0
2
2
0
Short-term forecasting Besides long-term forecasting, we also conduct experiments for short-term forccasting with 4 prediction longths (i.c., {12, 24, 48, 96}} on PEMS datasets as in SCINet (Liu et al. 2022a
and iTransformer (Liu et al. 2024). Full results on 4 PEMS datasets with 4 different prediction lengths are shown in Table 2 Generally, our model outperforms other baselines on all prediction lengths and all PEMS datasets, which demonstrates the superiority of capturing cross-channel cross-time relationships for short-term forecasting, Additionally, we observe that PatchTST usually underperforms iTrrasformer by a large margin, suggesting that modeling channel dependencies is necessary for PEMS datasets. The worse results of ITrrasformee, compared with our model, indicate that cross-channel temporal relationships are important and should be captured on these datasets.
5.2 Model Analysis
Ablation study We conduct the ablation study to verify the effectiveness of our dispatcher module by using the same setting (e.g., the number of layers, hidden dimensions, batch size) for comparing the our model with and without dispatchers. In Table 3 we can see that adding dispatchers helps to reduce GPU usage. In
Table 3: The effectiveness of our dispatcher module. OOM indicates the "Out of Memory" error on GPUs (we a single A100 GPU of memory 40GB).
ETTml
Weather
ECL
Traffic
MSE
Mem
MSE
Mem
MSE
Mem
MSE
Mem
w/o dispatchers
0.385
2.56GB
0.247
9.17GB
OOM
OOM
OOM
00M
w/ dispatchers
0.379
2.33GB
0.242
5.13GB
0.166
13.32GB
0.439
22.87GB
8

---

Published in Transactions on Machine Learning Research (06/2025)
ECL and Traffic, the version without dispatchers even leads to out-of-memory (OOM) issues. Moreover, we observe that the memory reduction becomes more significant when the number of variates increases. On ETTml with 7 variates, the memory only reduces from 2.56GB to 2.33GB, while on ECL and Traffic, it reduces from OOM (more than 40GB) to 13.32GB and 22.87GB, respectively.
The effect of different lookback lengths We also investigate how different lookback lengths would change the forecasting performance. With increased lookback lengths, we compare the forecasting performance of our model with that of several representative baselines in Figure The results show that, when using a relatively short lookback length (i.e., 48), our model generally outperforms other models by large margin. It suggests that model has more powerful learning ability capture the dependencies even with short lookback length, while other models usually require longer lookback lengths to provide good performance. Moreover, by increasing the lookback length, the performances of our model and PatchTST usually improve, whereas the performance of Transformer remains almost the same on ECL dataset.
Mean Squared Error vs Lookback Length on Weather
Mean Squared Error Lookback Length ECL
Transformer
0.26
0.40
PatchTST
irranfformer
0.24
Our Model
0.22
0.35
Transtormer
0.20
PatchTST
0.30
(Transformer
0.18
Our Model
0.25
0.16
0.20
0.15
48
96
192
336
720
48
96
192 Lookback Length
336
720
Lookback Length
Figure 5: Performance with different lookback lengths and fixed prediction length 96.
The effect of different patch sizes As we use patching in our model, we further examine the effect of different patch sizes. The patch size and the lookback length together determine the number of tokens for variate. In Figure 6 we demonstrate the performance by varying different patch sizes and lookback lengths. With lookback length of 64, the performance of using patch size 64 is much worse than that of patch size 8 Tt. indicates that, when the number of tokens of a variate is extremely small (i.e., only token for lookback length 64), the performance is not satisfactory as no enough fino-grained information. This could also be the reason why iTransformer may be not. ideal some cases use exactly a single token for a variate. Additionally, we observe that, for different lookback lengths, too small or too large patch size can lead bad performance. The reason may be that too many tokens or too less tokens would increase the difficulty of training.
The number of dispatchers In our model, propose to use several dispatchers to reduce the memory complex- ity with the number of dispatchers as a hyper-parameter. Here, we dive deep into the tradeoff between GPU mem- ory and MSE by varying the number of dispatchers. In Table 4 we demonstrate the performance and GPU mem- ory of different numbers of dispatchers on Weather and ECL with the prediction length 96. The results show that, with only 5 dispatchers, the performance is usually worse than with more dispatchers. It suggests that we should avoid using too few dispatchers as it may affect the model performance. However, with fewer dispatchers, the GPU memory usage is less as shown our complex- ity analysis in Section 4.1 For larger datasets like ECL, increasing the number of dispatchers leads to more signifi-
Weather
Lookback (64), Predict (96)
Lookbark (96), Prodict Length (96)
Lookback Predict Length (96)
Lookbark (336), Prodict Langth (96)
0.165
0.155
0.150
16
Patch
Figure 6: Performance with different patch sizes and lookback length.
9

---

Published in Transactions on Machine Learning Research (06/2025)
The Layer
The Last Layer
20.0
40
17.5
15.0
12.5
Frequency
25
10
2.5
0.0
0.094
0.096
0.098
0.100
0.102
0.104
0.106
0.108
0.0
0.1
0.2
0.3
0.4
0.5
Attention Weights
Attention Weights
Figure 7: The distributions of multiplied attention weights between two patch tokens on Weather.
cant memory increase, compared with the smaller dataset (i.e., Weather).
Table 4: The performance and GPU memory usage of varying dispatchers on Weather and ECL.
The number of dispatchers
5
10
20
50
MSE
0.1575
0.1552
0.1573
0.1566
Weather
GPU Memory (GB)
2.165
2.191
2.233
2.405
MSE
0.1348
0.1347
0.1343
0.1338
FCT.
GPU Memory (GB)
12.807
13.389
14.335
16.509
Attention Weights With our dispatcher module, we have two attention weights matrices, one from patch tokens to dispatchers and one from dispatchers patch tokens, with the size N l and k x N, respectively. multiplied attention matrix with the size N x N
Multiplying these two attention matrices gives that directly indicates the importance between two patch tokens. We demonstrate the multiplied attention weights from the first layer and the last layer in Figure 7 As shown, in the last layer, the distribution is visibly shifted to the left side, meaning that most of the token pairs have low attention weights, while fow token pairs have high attention weights. may suggest that the last layer indeed learns how to distribute the information to important tokens. In contrast, the first. layer has a more even distribution of attention weights, indicating that it distributes information more evenly to all tokens.
The importance of cross-variate cross-time depen- dencies With the multiplied altention weights, we fur- ther demonstrate the percentages of patch token pairs from different variables and different times for groups of patch tokens pairs with varied attention weights in Fig- ure We observe that the groups of patch token pairs with higher attention weights have a higher percentage of pairs from different variates and different times. For example, for all token pairs, the percentage is 87.50, while the percentage is 89.91 for top 0.5% token pairs with the highest allention weights. It suggests that more pairs of patch tokens with high attention weights come from dif- ferent variates and times. Therefore, effectively modeling cross-variate time is crucial for multivariate time series forecasting.
The percentage of patch taken pars from different vars and fiferent times
90.0
89.91
Pereenage
68.5
88:31
87.5
Figure 8: Patch token pairs with higher top atten- tion weights are more likely from different variates and different times.
10

---

Published in Transactions on Machine Learning Research (06/2025)
6 Case Studies
6.1 Visualization of Multivariate Correlations
To further investigate the ability of capluring multivariate correlations, in Figure we provide two cases of visualizations the correlation map of multivariate relationships in the predicted time series from Solar-Energy. We can find that, the correlation map of UniTST is similar to the correlation map of ground truth time series, which indicates that the variate dependencies are well-captured by UnITST. In contrast, compared with ours, the correlation map of iTransformer is less aligned with that of ground truth time series.
Correlations Ground Truth
Correlations UniTST's predictions
Correlations iTransformer's predictions
1.0
20
00
0.4
0.2
200
20
1.0
110
0.0
(a.
Correlations
Correlations UniTST'S predictions
Correlations ITransformer's predictions
1.0
0.8
0.6
130
0.0
300
120
20
125
20
100
(b)
Figure 9: The correlation maps of multivariate relationship with different models. The x-axis and y-axis both represent the variate index. The darker color indicates the stronger correlation.
7 Conclusion
In this work, first point out the limitation of previous works time series transformers for multivariate forocasting: their lack of ability to effectively capture intor-scries and intra-scrics dependencies simultancously. We further remonstrate that inter-series and intra-series dependencies are crucial for multivariate time series forecasting as they commonly exist in real-world data. To mitigate this limitation of previous works, we propose simple yet effective transformer model UniTST with dispatcher mochanism effectively capture inter-series and intra-series dependencies. The experiments on 13 datasets for time series forecasting show that our model achieves superior performance compared with many representative baselines.
11

---

Published in Transactions on Machine Learning Research (06/2025)
Moreover, conduct the ablation study to verify the effectiveness of our dispatcher mechanism and demonstrate the importance of inter-series and intra-series dependencies. Lastly, we also provide model analyses to demonstrate the importance of inter-series and intra-series dependencies, and specifically, a case study on visualization of multivariate correlations validates the ability of our model to capture variate (inter-series) dependencies. Our study emphasizes the necessity and eflectiveness of simultaneously capturing inter-variate and intra-variate dependencics in multivariate time series forecasting, and our proposod designs may represent step toward this goal.
References
Ebtesam Almazrouci, Hamza. Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Merouane Debbah, Etienne Goffinet, Daniel Heslow, Julien Launay, Quentin Malartic, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. Falcon-40B: an open large language model with state-of-the-art performance. 2023.
Nicholas Carlini, Milad Nasr, Christopher A Choquette-Choo, Matthew Jagielski, Trena Gao, Anas Awadalla, Pang Wei Koh, Daphne Ippolito, Katherine Lec, Florian Tramer, et al. Are aligned ncural networks adversarially aligned? Xrxiv preprinl arXiv:2306.15447 2023.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vieuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https://lmsys.org/blag
Abhimanyu Das, Weihao Kong, Andrew Leach, Rajat Sen, and Rose Yu. Long-term forecasting with tide: Time-series dense encoder. arXiv preprint arXiv:2304.08. 2023.
Google. An important next step our ai journey, 2023. URL https://bog.googll//cchhooo http://oooggee/ccoooo//////////a ai-search-updates/.
Lu Han, Han-Jia Ye, and De- De-Chuan Zhan. The capacity and robustness trade-off: Revisiting the channel independent strategy for multivariate time series forecasting. arXiv preprint arXiv:2304.05206, 2023.
Lu Han, Xu- Yang Chen, Han-Jia Ye, and De- Chuan Zhan. Softs: Efficient multivariate time series forecasting with series-core fusion. Xrxiv preprint arXin:2444.14 14.44444.1119 497, 2024.
Qihe Huang, Lei Shen, Ruixin Zhang, Shouhong Ding, Binwu Wang, Zhengyang Zhou, and Yang Wang. Cross- GNN: Confronting noisy multivariate time series via cross interaction refinement. In Thirty-seventh Confer- ence on Neural Information Processing Systems, 2023. URL
Romain Ilbert, Ambroise Odonnat, Vasilii Feofanov, Aladin Virmaux, Giuseppe Paolo, Themis Palpanas, and levgen Redko. SAMformer: Unlocking the potential of transformers in time scries forecasting with sharpness-aware minimization and channel-wise altention. In Forty-first International Conference on Machine Learning, 2024. URL
Sonain Jamil, Md Jalil Piran, and Oh-Jin Kwon. A comprehensive survey of transformers for computer vision. Drones, 7(5):287, 2023.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ICLR, 2015.
Guokun Lai, Wei- Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. SIGIR, 2018.
Jianxin Li, Xiong Hui, and Wancai Zhang, Informer: Beyond efficient transformer for long sequence time-series
forecasting. orXiv: 2012.07436, 2021.
Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer time series forecasting. NewrIPS, 2019.
Zhe Li, Shiyi Qi, Yiduo Li, and Zenglin Xu. Revisiting long-term time series forecasting: An investigation on linear mapping. Xix preprint ar arXi::2305.1072 2023.
12

---

Published in Transactions on Machine Learning Research (06/2025)
Minhao Liu, Ailing Zeng, Muxi Chen, Zhijian Xu, Qiuxia Lai, Lingna Ma, and Qiang Xu. Scinet: time series modeling and forecasting with sample convolution and interaction. NewrIPS, 2022a.
Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li, Weiyao Lin, Alex X Liu, and Schahram Dustdar. Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting. International conference learning representations, 2021a.
Yong Liu, Haixu Wu, Jianmin Wang, and Mingsheng Long. Non-stationary transformers: Rethinking the stationarity in time scries forecasting, NewrIPS, 2022b.
Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long. itransformer: Inverted transformers are effective for time series forecasting. In ICLR, 2024.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 10012 10022, 2021b.
Jiecheng Lu, Xu Han, Yan Sun, and Shihao Yang. CATS: Enhancing multivariate time series forecasting by constructing auxiliary time series as exogenous variables. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 32990 33006. PMLR, 21 27 Jul 2024.
Donghao Luo and Xue Wang. DeformableTST: Transformer for time series forecasting without over-reliance on patching. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL (oopperreiiew.eeeforrmmma.
MosaicM.. Introducing mpt-7b: A new standard for open-source, commercially usable 1lss, 2023. URL www.mosaicml..com /blgg/ mpt-7b. Accessod: 2023-05-05.
Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. time series is worth 64 words: Long-term forecasting with transformers. ICLR, 2023.
OpenAl. OpenAl: Introducing ChatGPT, 2022. URL
Adam Paszke, S. Gross, Francisco Massa, A. Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Z. Lin, N. Gimelshein, L. Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performant deep learning library. NewrIPS, 2019.
Hugo Touvron, Thibaut Lavril, Gantier Izacard, Xavier Martinet, Maric- Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, al. Llama: Open and efficient foundation language models. arXiv preprint Xiv::22302. (1099022119971 13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. Xix preprint arXi::2307.0900 00988888 2023b.
Yuxuan Wang, Haixu WI, Jiaxiang Dong, Guo Qin, Haoran Zhang, Yong Liu, Yun-Zhong Qiu, Jianmin Wang, and Mingsheng Long. Timexer: Empowering transformers for time series forecasting with exogenous variables. In The Thirtt-eighth Annual Conference on Neural Information Processing Systems, 2024. URL
https://openneviewwnnt/ /ttp://// www..ettoooumm/dd//AeeUUUUUOAA
Bichen Wu, Chenfeng Xu, Xiaoliang Dai, Alvin Wan, Peizhao Zhang, Zhicheng Yan, Masayoshi Tomizuka, Joseph Gonzalez, Kurt Keutzer, and Peter Vajda. Visual transformers: Token-based image represcntation and processing for computer vision. arXiv preprint arXiv:2006.00 006677000603677,,, 2020.
Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with Into-Correlation for long-term series forecasting. NeurIPS, 2021.
13

---

Published in Transactions on Machine Learning Research (06/2025)
Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. Timesnet: Temporal 2d-variation modeling for general time series analysis. TCI.R, 2023.
Guoqi Yu, Jing Zou, Xiaowei Hu, Angelica I Aviles-Rivcro, Jing Qin, and Shujun Wang. Revitalizing multivariate time series forecasting: Learnable decomposition with inter-series dependencies and intra-series variations modeling. arXiv preprint or Xix:24002. 1 2294,, 2024.
Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? AAAI, 2023.
Jiawen Zhang, Shun Zheng, Xumeng Wen, Xiaofang Zhou, Jiang Bian, and Jia Li. ElasTST: Towards robust
variod-horizon forecasting with elastic time-series transformer. In The Thirty-cighth Annual Conference Neural Information Processing Systems, 2024. URL http://wwwwii.eeeeffoommmddu/uua
Yunhao Zhang and Junchi Yan. Crossformer: Transformer utilizing cross-dimension dependency for multi- variate time series forecasting. ICLR, 2023.
Lifan Zhao and Yanyan Shen. Rethinking channel dependence for multivariate time series forccasting:
Learning from leading indicators. In The Twelfth Internalional Conference on Learning Representations, 2024. URL https:/ wtwpe//eeiiieeee/ooorroiiiiieeee
Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. FEDformer: Froquency enhancod decomposed transformer for long-term series forecasting. ICML, 2022.
A Diverse Cross- Time and Cross-Varias Dependencies
We further illustrate the cross-time cross-variate correlations on Exchange, Weather, ECL datasets in Figure I0 We can see that correlation patterns for different datasets are quite different. Additionally, even for a specific dataset with different variate pairs, the correlations of cross-variate patch pairs are also very diverse. For example, for Exchange, with variate pairs (1,3), the patches at the same time step are usually strongly correlated. In contrast, with variate pairs (3,4), the patches can sometimes even have zero correlation coefficient. Moreover, in Figure 10 for a specific dataset with a specific pair of variates (i.e., in a subfigure), we have similar observations as we discussed in Sec 3 that there is no consistent correlation pattern for different patch pairs of two variates and inter-variate dependencies are at the fine-grained patch level. These examples further demonstrate the ubiquity and the diversity of these cross-time cross-variate correlations in real data. This also justifies the motivation of this paper propose a better method to explicitly model cross-time and cross-variate vintra-variate and inter-variate) dependencies.
B Discussion on Computational Complexity
Moreover, we provide the computational complexity analysis of different models. As Feedforward networks in different models have similar complexities, we mainly analyze the computational complexity of the attention mechanism. For UniTST, the designed attention mechanism uses cross-attention with dispatchers to reduce the complexity. It results in the complexity as O(kNyp where & is the number of dispatchers, N the number of variates, and p is the number of patches within a variate. For iTransformer, it utilizes self-attention on the variate dimension, which leads to the complexity O(N2). Additionally, PatchTST uses self-attention the time dimension and treats each variate independently. As result, the complexity is O(Np²). We can see that different models have different advantages in different scenarios. For example, when handling data with a long time series but with fewer variates, ITransformer should be faster than others as it doesn't. 't depend on p. Comparing UniTST and PatchTST, when p is relatively small, then the complexity should be similar (we set & as 10 in our experiments). UniTST may be slower than iTransformrr when the length of the time series and the number of variates are both extremely large. For this extreme scenario, we leave further investigation for future work.
14

---

Published in Transactions on Machine Learning Research (06/2025)
Exchange vars 11,3)
Weather (0.10)
Electricity (50.100)
16
20.
33-
122
-01
(b)
(c)
Exchaange vars [3,4]
Weather vars (0.20)
Electricity vars (50,200)
20
24
34
26
(d)
(I)
Figure 10: Diverse cross-time time cross- cross-variate variate dependencies commonly exist in real data.
C
More on Experiments
C.1 Datasets
Following Liu et al. 2024), we conduct experiments on 13 real-world datasets to evaluate the performance of our model including (1) a group dalasets ETT (Li et al. 2021) contains factors electricity transformer from July 2016 July 2018. There are four datasets where ETTml and ETTm2 are recorded every 15 minutes, and ETTTh and ETTh2 recorded every hour; (2) Exchange (Wu et al. 2021 contains daily exchange rates from 8 countries from 1990 to 2016. (3) Weather (Wu et al. 2021) collects the every 10-min data of 21 meteorological factors from the Weather Station of the Max Planck Biogeochemistry Institute in 2020. (4) ECL (Wu et al. 2021) records the electricity consumption data from 321 clients every hour. (5) Traffic (Wu et al. 2021) collects hourly road occupancy rates measured by 862 sensors of San Francisco Bay area freeways from January 2015 to December 2016. (6) Solar- Ennegy (Lai et al. 2018) records the solar power production of 137 PV plants in 2006, which are sampled every minutes. (7) group of datasots PEMS records the public traffic network data in California and collected by 5-minute windows. We the same four public datasets (PEMS03, PEMS04, PEMS07, PEMS08) adopted in SCINet (Liu et al. 2022a) and iTransformer [Liu et al. 2024). We provide the detailed dataset statistics and descriptions in Table 5
We also use the same train-validaaiion-tes splits as in TimesNet (Wu et al. 2023) and iTransformer (Liu et al. 2024). For the forecasting setting, following i'Tansformer (Liu et al. 2024). we use the fixed lookback length as 96 all datasots. terms of the prediction lengths, we use the varied prediction lengths in (96,
192, 336, 720) for ETT, Exchange, Weather, ECL, Traffic, Solar-Enee Shar-Energy. For PEMS datasets, we use the prediction lengths (12, 24, 48, 96) for short-term forecasting.
15

---

Published in Transactions on Machine Learning Research (06/2025)
Table 5: Detailed dataset statistics. # variates denotes the variate number of each dataset. Dataset Size denotes the total number of time points in (Train, Validation, Test) split respectively. Frequency indicates the sampling interval of data points.
Dataset Name
# variates
Prediction Length
Dataset Size
Frequency
Information
ETTh1, ETTh2
(96, 192, 336, 7203
(8545, 2881, 2881)
Hourly
Electricity
ETTml, ETTm2
(96, 192, 336, 720)
(34465, 11521, 11521)
15min
Electricity
Exchange
(96, 192, 336, 7203
(5120, 665, 1422)
Daily
Economy
Weather
21
[96, 192, 336, 7203
(36792, 5271, 105400
10min
Weather
ECL
321
(96, 192, 336, 7203
(18317, 2633, 5261)
Hourly
Electricity
Traffic
862
336, 7203
(12185, 1757, 3509)
Hourly
Transportation
Solar-Energy
137
[96, 192, 336, 7203
(36601, 5161, 10417)
10min
Energy
PEMS03
358
(12, 24, 48, 96)
(15617, 5135, 5135)
5min
Transportation
PEMS04
307
(12, 24, 48, 96)
(10172, 3375, 3375)
5min
Transportation
PEMS07
883
[12, 24, 48, 96}
(16911, 5622, 5622)
5min
Transportation
PEMS08
170
(12, 24, 48, 96)
(10690, 3548, 3548)
5min
Transportation
C.2
Experimental Setting
We conduct all the experiments with Pylorrc Paszke et al., 2019 and utilize a single NVIDIA A100 GPU with 40GB memory. We describe the hyperparameter choices used in our experiments the following. For the optimizer,
2015) with the learning rale in {10-3, x 10-1, 10-4). The batch sizes are selected from {16, 32, 64, 128) depending on the dataset sizes. The maximum mumber of also use the early stop strategy to stop the training when the loss does not decrease in 10 epochs. The number of layers of our Transformer blocks is selected from (2,3.4). The hidden dimension of D set from {128, 256, 512]-
use ADAM (Kingma & Ba
training epochs is set to 100 as in Nie et al. 2023). Meanwhile,
For the experimental results of our model, we report the averaged results with 5 runs with different seeds. For the results of previous models, we reuse the results from iTrrnsformmr paper (Liu al. 2024) we are using the same experimental setting.
C.3 Additional Experimental Results
C.3.1 Full Result of Forecasing
Due to the space limitation, we only display the averaged results over 4 prediction lengths for datasets on long-term forecasting. Here, we provide the full results of long-term forecasting in Table G In summary, our model achieves the best results on 24 and 26 out of 36 settings with different prediction lengths among other baselines.
C.3.2 The correlation maps of multivariate relationship on different datasets
In this section, we provide additional visualizations on the correlationship map of multivariate relationship for ETTml dataset. in Figure 11 Similar to Figure for Solar-Fnergy, also shows that the variate dependencies are well-captured by UnTTST.
16

---

Published in Transactions on Machine Learning Research (06/2025)
Table 6: Full results of the long-term forccasting task. We compare extensive competitive models under different prediction lengths following the setting of TimesNet (2023). The input sequence length is set to 96 for all baselines. Aug means the average results from all four prediction lengths.
Models
UnITST (Ours)
SOFT [024
FTrandormer [2025]
RLinea
PSteTTST
Crossoemmr
TIDE 2023
TimesNet
DLincot
SCINet
FEDioormr
Stationary
Autoformer (202]
2023
2023
2023]
2023
2023
20226
2022
20226
Metric
MSB
MAE
MSR
MAE
MSK
MAE
MSE
MAR
MSE
MAR
MSE
MAR
MSE
MAR
MSE
MAR
MSE
MAR
MSB MAR
MSE
MAR
MSK
MAR
MSK
MAK
96
0.318
0.352
0.325
0.361
0.331
0.368
0.335
0.376
0
0.307
0.404
0.426
10,354
0.387
0.387
0.333
0.375
0.375
0.345
0.372
0.438)
0.374
01.38 1.335s....0 1.503 0.475
00.350
0.380
0.375
0.389
0.377
0.391
0.391
0.392
0.307
0.385
0.450
0.451
0.338
0.404
0.374
0.357
0.380
0.38900.439
0.439
0.450
0.426
0.44)
0.459
0.444
0.553 0.495
398
0.396
0.404
0.205
0.412
0.126
0.420
0.424
0.415
0.009
0.410
0.692
0.510
0.425
0.425
0.410
0.413
0.413 0.413 0.440 0.485
0.44G
0.44500.495
01.354
(.621 0.h37
720
00.449
0:440
0.465
0.447
0.491
0.450
0:487
0.450
0.464
0.480
0.660
0.689
0.487
0.461
0.478 0.450
0.45000.474
0.474
0.455
0.550
0.543
0.490
0.516 0.561
Arg
[0:379
0.394
0.395
0.403
0..077
0.410
0:414
0.007
0.367
0.400
0.513
0.406
0.419
0.419/0.000
0.000
0.636
0.607/o.
0..85
0.881
0.448
0.452/0.081
0.456|0.588
0.517
BTT1m2
0.176
112002
1.180
03.261
0.388
012264
0.382
01266
(0.1/76
0.269
0.287
0.3000
0..007
0.335
0.187
0.25770.1333
0.29210.220
0.377
0.200
01287/1
0.274 (1225 1.339
192
0.243
0.304
0.245
0.306
0.250
0.309
0.246
0.304
0.241
0.302
0.414
0.402
0.290
0.364
0.249
0.329
0.2299
0.284
0.382 0.309
0.382
0.435
0.200
0.322
0.250
0.339 0.281 0.340
0.902
0.343
1.319
0.862
0.311
0.307
11:112
01344
0.507
0.542
0.357
0.423
1.223 0.353
1.a00 0.427 0.Ga7
0.533
0.322
0.300
0.334
0.363 (1339 11372
720
0.398
0.395
0.405
0.401
0.412
0.407
0.407
0.398
0.402
0.400
1.730
1.042
0.538
0.524
0.408
0.403
0.005
0.554
0.522
0.730
0.421
0.415
0.417
0.413 0.43 0.432
Avg
|0.280
0.326
0.2257
0.330
0.255
0.332
0.286
0.327
0.281
0.826]
0.757
0.610
0.335
0.404
(02293 0.335/0.350
0.001 (0.77] 0.637]
0.300
0.340|0.3
0.347/0 (0.33777 0.371
0.363
0.598
0.3881
0.300
0.586
0.386
0.5951
0.414
0.419
0.425
0.448
0,802 0.400/0.055
0.54 0.500
0.419/0.515
0.091 (0,009 0.159
192
0.434
0.425
0.435
0.431
0.441
0.436
0.437
0.424
0.400
0.445
0:471
0.474
0.525
0.492
0.436 0.
0.429.0.337 0.
0.719 0.631 0.420
0.719
0.448
0.534
0.504 0,500 0.082
0.471
0.445
0.680
0.452
0.066
0.086
0.570
0.546
0.565
0.515 0.001
0.681
0.450
0.0500.4500
0.465
0.588
0.535 0.521 0.096
720
0.447
0.489
0.499
0.488
0.503
0.491
0.481
0.470
0.500
0.488
0.653
0.621
0.594
0.558
(.521
0.500
0.511 0.519 0.510
0.516 0.836 0.699
0.500
0.507 (.663
0.616 0.512
Aeg
0.442
1.485
[1.049
0.442
0.454
0.407
0.446
0.4344
0.469
0.454
10.229
0.522
(0.43 0.507
0.450|u..00
0.647/o
0.480/0..500
0.537
0.537/a..00
(0.006 01487
96
0.299
0.342
0.207
0.347
0.2297
0.3:0
(0.288
0.333
0.302
0.348
0.745
0.584
0,,000
0.40000.3000
0.300
0.374|0.333
lo.333
0.387/0.707
0.62/
[03358
0.597
3.3770.6500
0.476
0.458/o 0.366 0.388
111222
192
0.370
0.390
1.973
0.384
0.390
0.400
0.354
0.390
0.388
0.400
0.877
0.650
0.522
0.509
0.402
0.414
0.477
0.8600.860
0.860
0.689
0.422
0.512
0.40330.454 0.452
336
00.822
0.408
0.610
0.426
0.128
0.432
0:416
0.126
0.426
0.433
1.043
0.731
0.643
0.57)
0.452
0.552..500
0.452
0.504
0.541
1.000
0.744
0.406
0.487
0.552 0.55| 0.682 0.986
720
00.409
0.483
[0:011
034333
0.427
0.445
0.420
0.440
0.431
0.446
1.104
0.763
0.874
0.079
0.402
0.488
0.831
0.657
1.249 0.838
0.463
0.474 0.502
0.500/0.515 0.511
Arg
|0.366
0.393
0.375
0.400
0.383
0.407
0:374
0.396
0.367
0.607
00.912
0.684
0.611
0.530
0.114
0.427
0.027/0.530
(0.59
0.515
0.515/0.944
0.954
0.725
0.437
0.449
0.449
0.516|0..50 0.459
96
10.139
012255
1.143
0.233
0.148
11:0001
102001
01281
0.181
0.270
0.219
0.314
0.297
0.325
1.32990.16
1.188
0.272
0.272/0.197
(.197
0.242
0.340
0.300
0.30s
0.109
0.275
1.317
ECL
0.1.5
1.270
1.108
0.248
0:501
0.283
0.274
0:231
0.322
0.296
0.390
0.184
0.289
0.196
0.250
0.257
0.350
0.315
0.182
0.286
0.384
398
0.170
0.268
1.178
0,200
0.17%
112661
0.216
0.298
0.214
0.295
0.246
0.317
0.249
0.364
0.198
0.3000.200
0.300
0.209
0.201
0.299
0.209
0.214
0.329
0.200
0.304 0.251 1.308
0.1188
0.208
0.218
0.305
0:257
0.331
0.246
0.324
0.2280
0.363
0.373
0.320
0.245
0.333
0.230
0.353
0.222
0.323 (.255 0.361
Avg
[0.106
0.202
0.174
0.264
0.178
0.270
0:219
0.295
0.2005
012250
00:244
0.333
|0,253
0.344/0.
0.1512
0.2200
(0.212
0.300
0.307
0.224
(1327
11193 0.290/o.222 0.338
Exchange
96
10.080
0.198
0.096
0.206
10.000
0.217
0.008
0.205
0.250
0.367
00.004
0.218
101
0.254
0.218/0.207
0.390
0 0.144
0.278/0.1
0.11 o 0.237/0.197 0.325
0.173
0.296
0.177
0.184
0.307
0.170
0.209
0.470
0.509
0.226
0.364
0.1 0,315
0.351
0.315
0.219 0.335 0,300
333
0.314
1..05
0.331
0.417
0:351
0.432
0.301
0.397
1.268
0,883
0,349
0.433
0.307
0.453
0.313
0.427
1.324
0.853
0.400
0.427
0.421
0.470..550 0.521
0.838
0.093
0.847
0.091
0.586
0.714
0.901
0.714
1.767
1.008
0.852
0.696
0.964
0.736
0.809
0.695
1.058
0.707
1.196
0.605
1.092
0.789 L.477 0.041
[0.351
0.398
0.3300
0.378
0.417
0..67
0.404
00.540
0.707
(0.370 0.01:/0..1.
1.336 0.603
0.014/u.770
(1.70
0.620)
0.529
0.420/0.00
0.054/0 (1.638 1..39
96
0.402
0.255
0.376
0.251
0.595
0.208
0.6449
0.589
0.462
0.205
0.522
0.290
0..05
0.403/0.500
0.503
0.321 1..050
3.788
0.180
0.587
0.338|0.013 0.388
Truffic
192
0.426
0.288
0.398
0.261
0.417
0.276
0.501
0.366
0.406
0.23%
0.630
0.293
0.736
0.474
0.617
0.535
0.50s 0.870
0.60r
0.604
0.373
0.618
0.340 0.382
996
0.449
0.275
0.411.
0.260
0.433
0.283
0.600
0.900
0.482
0.504
0.658
0.305
0.702
0.477
0 0.396.0000
0.
0.508
0.621
0.383
0.018 0.328 32880.02 0.022 0.337
720
0,460
0.
0.447
0.287
0.667
0.302
0.647
0.387
0.514
0.322
0.569
0.328
0.719 0.600
0.6 0.000
0.550
0.
0.655
0.504
0.8.11
0.525
0.620
0.382
0.653 0.355 o. 0.680 0.008
Ang
0.274
00.400
0.267
0.282
(06066
0.578|
0.461
0.304
|0.500
0.304
|0.700
0.336
0.385jo.800
0.500/06610
0.600
0.376|0.620
0.04 0.310jo.822 0.379
Weather
96
10.156
0.202
0.108
0.208
0.174
0.214
0.199
0.292
0.177
0.218
0.108
0.230
10.202
0.281
10.12
0.220/0.196
0.253/0.22
0.306
0.217
0.29610.173
0.22310.268 0.338
192
0.207
0.250
11.217
0.255
00.221
11226
0:240
11277
0.226
0.250
0.200
0.277
0.242
0.299
0.210
0.2553
1.297
0.2960.2661
0.330
0.240 0.250 0.307 1.367
333
0.263
0,292
0.282
0.300
0.278
0.296
0.299
0.307
0.278
0.297
0,272
0.335
0.287
0.333
0.280
0.253
0,303
0,309
0.330
0.330
0.921
0.338 0,309 0.395
720
00.340
0.341
11.366
0.361
0..358
0.364
0.354
0.364
0.348
0.208
0.418
0.353
0.350
0.391
0.427
0.400
0.425
0.014
0.410/0.319 11:28
Avg
10.241
0.271
0.225
0.278
0.2258
0.278
0.227
0.201
|0
0:225
0.251
|0.350
0.315
|0.271 0.320|0.250
0.257
2...2777700 |0.205
0.317/0.200
0.305/0
00000
0.300/0.220
0.31400.33 0.382
Solar-Caenge Caergy
10.189
0.228
0.200
0.230
0.203
0.237
0.322
0.039|
0:234
0.236
0.310
0.301
10.312
0,399 0.250 0. 0.292/0.290 2.200
0.375
0,344
0.242
0.342/0.213
0.249/0.882 0.711
192
00.222
0.253
0.220
0.255
0.233
0.261
0:330
0.356
0.267
0.310
0.734
0.725
0.339
0.416 0.296 0.216..296
0.918 0.386
0.518
0.380
0.288
0.390/
0.272 0.834 0.092
0.244
0.275
1.245
0.266
0.248
0.273
0.297
0.389
0.200
2200
0.315
0.750
0.7a5
o.
0.319
0.353
0.315
0.389
0.376
0.296
0.941 0.723
Solar-
720
0.247
0.282
0.245
0.277
0.219
0.275
0:397
0.356
0.317
0,780
0,705
0.300
o. 0.538
0.537
0.356 0. 0.300 0.368
0:357
0.427
0.
0.285
0.295
1..50.... 0.882 0.717
Aug
10.225
012900
11,229
0.230
0.233
0.262
0.309
0.306
0.277
0.307
00.01
0.629
|0.347
0.417 (1.0000000 0.015/0.330 0.a01 (.
0.370/0.221
0.2991
0.381
(0.261
0.381 (0.850 1.711
(Comt)
80
27
11
5
5
0
0
0
0
17

---

Published in Transactions on Machine Learning Research (06/2025)
Correlations Ground Truth
Correlations UnITST's predictions
Correlations
ITransformer's predictions
1.0
0.8
0.6
0.4
(a)
Correlations Ground Truth
Correlations UniTST's predictions
Correlations
ITransformer's predictions
0.8
0.6
0.4
0.2
(b)
Figure 11: The correlation maps of multivariate relationship with different models on ETTm1. The x-axis and y-axis both represent the variate index. The darker color indicates the stronger correlation.
18


Reviews:
[{"review_id": "tQYSct3zdy", "reviewer": "Reviewer_RSTB", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n\n1. The paper discusses the limitations of previous Transformer-based methods for multivariate time series forecasting (inability to capture both inter-variable and intra-variable relationships simultaneously) and validates this viewpoint through examples.\n2. UniTST effectively captures both inter-variable and intra-variable dependencies by flattening all patches from different variables into a unified sequence, and reduces memory overhead through the dispatches module. Ablation study demonstrate the effectiveness of each module.\n3. Extensive experiments on multiple real-world datasets have demonstrated the effectiveness of the method.\n\nCons:\n\n1. In the rapidly evolving field of time series forecasting, it is insufficient to only use iTransformer as a baseline. Further comparisons with more advanced methods (in 2024 or later) are needed to demonstrate the effectiveness of the approach, such as Samformer [1], ElasTST [2], and DeformableTST [3].\n2. Although the effectiveness of the dispatcher module has been explored, demonstrating its ability to effectively reduce memory consumption, an analysis of the model's efficiency remains essential.\n3. As seen in Table 4, there is no clear relationship between the number of dispatcher modules and the MSE performance across different datasets. Does this imply that the number of dispatcher modules is a sensitive hyperparameter, requiring fine-tuning across different datasets to identify the most suitable value?\n4. UniTST captures both inter-variable and intra-variable dependencies by flattening all patches into a unified sequence. However, different datasets have varying degrees of dependence between variables and within variables. For example, datasets like ECL, which have a large number of channels, may require a greater emphasis on capturing inter-variable relationships. It is necessary to further complement the ablation experiments to evaluate the performance of UniTST compared to models that only model intra-variable relationships or inter-variable relationships on different datasets.\n5. For the inter-variable relationship visualization in Figure 9, it is necessary to provide results on additional datasets to demonstrate the effective capture of inter-variable relationships by UniTST.\n\n[1] Samformer: Unlocking the potential of transformers in time series forecasting with sharpness-aware minimization and channel-wise attention. ICML, 2024.\n\n[2] ElasTST: Towards Robust Varied-Horizon Forecasting with Elastic Time-Series Transformer. NeurIPS, 2024.\n\n[3] DeformableTST: Transformer for Time Series Forecasting without Over-reliance on Patching. NeurIPS, 2024.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes UniTST for time series forecasting, which can simultaneously capture both inter-variable and intra-variable dependencies. Experiments on multiple real-world datasets demonstrate that UniTST achieves superior performance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "As in Cons.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "UVeRijoDVL", "reviewer": "Reviewer_yYJE", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nInnovation: The unified attention mechanism and dispatcher module present a novel solution to the challenge of modeling both inter-variate and intra-variate dependencies simultaneously.\n\nTheoretical Foundation: The introduction of the cross-time cross-variate correlation coefficient provides a theoretical basis for the diversity of dependencies, supporting the model's design.\n\nWeaknesses:\n\nInterpretability: While the attention weights are visualized to illustrate dependency relationships, the paper lacks a detailed explanation of how the dispatcher allocates information. A deeper analysis of the mechanism's workings would strengthen the study.\n\nBaseline Comparison: The paper does not compare with recent related work, such as SOFTS: Efficient Multivariate Time Series Forecasting with Series-Core Fusion (NeurIPS 2024). Additional experiments are needed to further validate the model's superiority.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a novel Transformer model named UniTST for multivariate time series forecasting (MTSF). The key contributions include:\n\nProblem Analysis: The authors identify the limitations of existing Transformer models in simultaneously capturing inter-variate and intra-variate dependencies, and validate the importance of these dependencies through real-world data.\n\nModel Design: To address the increased computational cost, the authors propose a unified attention mechanism that flattens patches from different variates into a single sequence to directly model cross-variate and cross-time dependencies. Additionally, a dispatcher module is introduced to reduce computational complexity from quadratic to linear, significantly improving scalability.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Additional Comparative Experiments: Include comparisons with state-of-the-art methods like SOFTS to comprehensively demonstrate the model's advantages.\n\nEnhanced Interpretability: Provide case studies or theoretical analysis to clarify how the dispatcher selects and prioritizes dependencies.", "broader_impact_concerns": "The paper does not raise significant ethical concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "y3x8YYeCt9", "reviewer": "Reviewer_6jKS", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n1. The unified attention mechanism offers a simple yet effective adaptation to existing multivariate Transformer models, enabling comprehensive modeling of complex dependencies.\n2. The paper includes visual explanations (Figures 1 and 2) that effectively communicate the model architecture and underlying concepts.\n3. The dispatcher mechanism significantly reduces computational complexity compared to full pairwise interactions across all variates.\n4. Comprehensive experiments across 13 diverse datasets demonstrate consistent performance improvements over state-of-the-art models like iTransformer, particularly for lookback length of 96.\n\nLimitations: \n1. The performance advantage over iTransformer diminishes with larger lookback lengths (as shown in Figure 5), suggesting that UniTST may not improve upon the state-of-the-art when using optimal lookback lengths.\n2. The argument that time-wise and variate-wise attention must be integrated within the same mechanism could be more thoroughly substantiated, as alternating layers of time-wise and variate-wise attention might theoretically achieve similar information flow.\n\nSuggestions for Improvement: \n1. Include computational efficiency comparisons (training times) between UniTST, Crossformer, and iTransformer to better highlight the practical benefits of the dispatcher mechanism.\n2. Explore the interpretability aspects of the dispatcher module to potentially extract meaningful patterns about time series relationships.\n3. Consider investigating a more adaptive architecture where attention is primarily used in the time direction, with variable interactions occurring only when necessary through a dynamic dispatcher, which could further improve scalability for massive but sparsely interacting systems.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces UniTST, a transformer-based architecture for multivariate time series forecasting that effectively addresses both inter-series and intra-series dependencies using a unified attention mechanism. The approach employs a dispatcher module with fixed query tokens to maintain efficiency when handling numerous variates. The model processes flattened patch tokens to simultaneously capture temporal and variate relationships, overcoming limitations in previous architectures.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see the weaknesses.", "broader_impact_concerns": "The paper presents a clean, efficient attention mechanism for multivariate time series forecasting that provides meaningful improvements over existing approaches, particularly with shorter lookback windows. The unified approach to capturing dependencies and the dispatcher mechanism represent valuable contributions to the time series and data mining field.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers positively acknowledge the significance and clarity of the paper. Reviewer yYJE highlights the paper’s comprehensive revisions, notably the inclusion of comparisons with state-of-the-art methods such as SOFTS and improvements in the interpretability of the dispatcher module through detailed attention visualizations. Reviewer 6jKS similarly appreciates the clarity, innovation, and empirical robustness of the proposed method. Reviewer RSTB, while leaning slightly towards acceptance, expresses mild reservations regarding the authors’ handling of baseline comparisons, noting that several suggested baselines were not fully incorporated. Despite these concerns, the reviewer concedes that the authors have adequately addressed major issues, bringing the paper to a technically sound and publishable state.

The revisions have sufficiently addressed the reviewers' core concerns, particularly regarding benchmark comparisons and interpretability.
Decision: Accept (poster)