Crossformer: Transformer Utilizing Cross-Dimension Dependency for Multivariate Time Series Forecasting
Authors: Yunhao Zhang
Junchi Yan
Venue: ICLR 2023 notable top 5% (2023)
Keywords: Transformer
multivariate time series forecasting
deep learning

Abstract: Recently many deep models have been proposed for multivariate time series (MTS) forecasting. In particular, Transformer-based models have shown great potential because they can capture long-term dependency. However, existing Transformer-based models mainly focus on modeling the temporal dependency (cross-time dependency) yet often omit the dependency among different variables (cross-dimension dependency), which is critical for MTS forecasting. To fill the gap, we propose Crossformer, a Transformer-based model utilizing cross-dimension dependency for MTS forecasting. In Crossformer, the input MTS is embedded into a 2D vector array through the Dimension-Segment-Wise (DSW) embedding to preserve time and dimension information. Then the Two-Stage Attention (TSA) layer is proposed to efficiently capture the cross-time and cross-dimension dependency. Utilizing DSW embedding and TSA layer, Crossformer establishes a Hierarchical Encoder-Decoder (HED) to use the information at different scales for the final forecasting. Extensive experimental results on six real-world datasets show the effectiveness of Crossformer against previous state-of-the-arts.

Published as a conference paper at ICLR 2023
CROSSFORMER: TRANSFORMER UTILIZING CROSS- DIMENSION DEPENDENCY FOR MULTIVARIATE TIME
SERIES FORECASTING
Yunhao Zhang & Junchi Yan
MoE Key Lab of Artificial Intelligence, Shanghai Jiao Tong University and Shanghai AI Lab (zhangyurhao, e j. Code: https:/ http///ttttt thub. com com/Think/ ab- Btttaaatr///rraaarromsss
edu.
ABSTRACT
Recently many deep models have been proposed for multivariate time series (MTS) forecasting. In particular, Transformer- models have shown great potential because they can capture long-term dependency. However, existing Transformer- based models mainly focus on modeling the temporal dependency (cross-time dependency) yet often omit the dependency among different variables (cross- dimension dependency), which is critical for MTS forecasting. To fill the gap, we propose Crossformer, a Transformer-basee dency for MTS forecasting. In Crossformer, the input MTS is embedded into 2D vector array through the Dimension- SemenntionSSemment Wise (DSW) embedding to preserve time and dimension information. Then the Two-Stage Attention (TSA) layer is proposed to efficiently capture the cross-time and cross-dimension dependency. Utilizing DSW embedding and TSA layer, Crossformer establishes Hierarchical Encoder- Decoder (HED) to use the information at different scales for the final forecasting. Extensive experimental results on six real-world datasets show the effectiveness of Crossformer against previous state-of-the-arts
model utilizing cross-dimension depen-
INTRODUCTION
Multivariate time series (MTS) are time series with multiple dimensions, where each dimension represents specific univariate time series (e.g. a climate feature of weather). MTS forecasting aims to forecast the future value of MTS using their historical values. MTS forecasting benefits the decision-making of downstream tasks and widely used in many fields including weather (Angryk et al., 2020), energy (Demirel et al., 2012), finance (Patton, 2013), etc. With the development of deep learning, many models have been proposed and achieved superior performances in MTS forecasting (Lea et al., 2017; Qin et al., 2017; Flunkert et al., 2017; Rangapuram et al., 2018; Li et al., 2019a; Wa et al., 2020; Li et al., 2021). Among them, the recent Transformer-basee models (Li et al., 2019b; Zhou al., 2021; Wu et al., 2021a; Liu al., 2021a; Zhou et al., 2022; Chen et al., 2022) show great potential thanks to their ability to capture long-term temporal dependency (cross-time dependency).
Besides cross-time dependency, the cross-dimension dependency is also critical for MTS forecasting, Le. for a specific dimension, information from associated series in other dimensions may improve prediction. For example, when predicting future temperature, not only the historical temperature, but also historical wind speed helps to forecast. Some previous neural models explicitly capture the cross-dimension dependency, i.e. preserving the information of dimensions in the latent feature space and using convolution neural network (CNN) (Lai al., 2018) or graph neural network (GNN) (Wu al., 2020; Cao et al., 2020) capture their dependency. However, recent Transformer-basee models only implicitly utilize this dependency by embedding. In general, Transformer- based models embed data points in all dimensions the same time step into feature vector and try capture dependency among different time steps (like Fig. (b)). In this way, cross-time dependency well captured, but cross-dimension dependency not, which may limit their forecasting capability.
"Junchi Yan is the correspondence author. This work was in part supported by NSFC (61972250, U19B2035, 62222607) and Shanghai Municipal Science and Technology Major Project (20210HZD7X0100))

---

Published as a conference paper at ICLR 2023
To fill the gap, propose Crossformer, Transformer-basee model that explicitly utilizes cross- dimension dependency for MTS forecasting. Specifically, we devise Aamension-SSement Wise (DSW) embedding to process the historical time series. In DSW embedding, the series in each dimension is first partitioned into segments and then embedded into feature vectors. The output of DSW embedding is a 2D vector array where the two axes correspond time and dimension. Then we propose the Sto---ttttAtttttionn (TSA) layer to efficiently capture the cross-time and cross-dimension dependency among the 2D vector array. Using DSW embedding and TSA layer, Crossformer establishes Hierarchical Sncoder-Decoder (HED) for forecasting. In HED, each layer corresponds scale. The encoder's 's upper layer merges adjacent segments output by the lower layer to capture the dependency at a coarser scale. Decoder layers generate predictions at different scales and add them up as the final prediction. The contributions of this paper are:
1) We dive into the existing Transformer-based models for MTS forecasting and figure out that the cross-dimension dependency is not well utilized: these models simply embed data points of all dimensions at a specific time step into a single vector and focus capturing the cross-time dependency among different time steps. Without adequate and explicit mining and utilization of cross-dimension dependency, their forecasting capability is empirically shown limited.
2) We develop Crossformer, a Transformer model utilizing cross- doos--iimnnioo dependency for MTS forecasting. This is one of the few transformer models (perhaps the first to our best knowledge) that explicitly explores and utilizes cross-dimension dependency for MTS forecasting.
3) Extensive experimental results six real-world benchmarks show the effectiveness of our Specifically, Crossformer ranks top- among the 9
Crossformer against previous models for comparison on 36 out of the 58 settings of varying prediction lengths and metries and ranks 2 on 51 settings.
2 RELATED WORKS
Multivariate Time Series Forecasting. MTS forecasting models can be roughly divided into statistical and neural models. Vector auto-regressive (VAR) model (Kilian & LAtkepohl, 2017) and Vector auto-regressive moving average (VARMA) are typical statistical models, which assume linear cross-dimension and cross-time dependency. With the development of deep learning, many neural models have been proposed and often empirically show better performance than statistical ones. TCN (Lea et al., 2017) and DeepAR (Flunkert et al., 2017) treat the MTS data as a sequence of vectors and use CNN/RNN to capture the temporal dependency. LSTTet (Lai et al., 2018) employs dependency and RNN for cross-time dependency. Another category
CNN to capture cross-dimense of works use graph neural networks (GNNs) to capture the cross-dimension dependency explicitly for forecasting (Li et al., 2018; Yu al., 2018; Cao et al., 2020; Wu et al., 2020). For example, MTGNN (Wu et al., 2020) uses temporal convolution and graph convolution layers to capture cross- time and cross- cooss-dimension dependency. These neural models capture the cross-time timse dependency through CNN or RNN, which have difficulty in modeling long-term dependency.
Transformers for MTS Forecasting. Transformers (Vaswani al., 2017) have achieved success in natural language processing (NLP) (Devlin et al., 2019), vision (CV) (Dosovitskiy et al., 2021) and speech processing (Dong et al., 2018). Recently, many Transformer-basse based models have been proposed for MTS forecasting and show great potential (Li et al., 2019b; Zhou et al., 2021; Wu et al., 2021a; Liu et al., 2021a: Zhou et al., 2022; Du et al., 2022). LogTrans (Li et al., 2019b) proposes the LogSparse attention that reduces the computation complexity of Transformer from O(L") to O (Tollog T.) Informer (Zhou et al., 2021) utilizes the sparsity of attention score through KI. divergence estimation and proposes ProbSparse self-attention which achieves O(L log L) complexity. Autoformer (Wu al., 202 introduces a decomposition architecture with an Auto-Correlation mechanism to Transformer, which also achieves the ( log L) complexity. Pyraformer (Liu et al., 2021a) introduces a pyramidal attention module that summarizes features at different resolutions and models the temporal dependencies of different ranges with the complexity of O(L). FEDformer (Zhou et al., 2022) proposes that time series have sparse representation frequency domain and develop a frequency enhanced Transformer with the (L) complexity. Preformer (Du et al., 2022) divides the embedded feature vector sequence into segments and utilizes segment-wing wise correlation based attention
In this paper, the meaning of complexity refers to both time and space overhead and they are often the same. This is also in line with the presentation in existing works (Wu et al., 2021a; Liu al., 2021a).
2

---

Published as a conference paper at ICLR 2023
0208
5.08
(a)
Figure 1: Illustration for our DSW embedding. Self-attention scores from a 2-layer Transformer trained on ETTh1, showing that MTS data tends to be segmented. (b) Embedding method of previous Transformer-basse models (Li et al., 2019b; Zhou al., 2021; Wu et al., 2021a; Liu et al., 2021a): data points in different dimensions at the same step are embedded into a vector. (c) DSW embedding of Crossformer: in each dimension, nearby points over time form a segment for embedding.
for forecasting. These models mainly focus reducing the complexity of cross-time dependency modeling, but omits the cross-dimension dependency which critical for MTS forecasting.
Vision Transformers. Transformer is initially applied to NLP for sequence modeling, recent works apply transformer to CV tasks to process images (Dosovitskiy et al., 2021; ; Touvron et al., 2021; Liu al., bb; Chen et al., 2021; Han et al., 2021). These works achieve state-of-the-an performance on various tasks in CV and inspire our work. ViT (Dosovitskiy et al., 2021) is one of the pioneers of split image into non-overlapping medium-sized
vision transformers. The basic idea of ViT patches, then it rearranges these patches into sequence to be input to the Transformer. The idea of partitioning images into patches inspires our DSW embedding where MTS is split into dimension- wise segments. Swin Transformer (Liu et al., 2021b) performs local attention within a window to reduce the complexity and builds hierarchical feature maps by merging image patches. Readers can refer to the recent survey (Han et al., 2022) for comprehensive study on vision transformers.
3 METHODOLOGY
In multivariate time series forecasting, one aims to predict the future value of time series XT+1.7-r E RTXD given the history X1.7 E R where 7 is the number of time steps in the future and past, respectively?. D > is the number of dimensions. A natural assumption is that these D series are associated (e.g. climate features of weather), which helps to improve the forecasting accuracy. To utilize the gross-dimension dependency, in Section 3.1, we embed the MTS using Dimension-Segment- Segment- Wise (DSW) embedding. In Section 3.2, propose Two-Stage Attention (TSA) layer to efficiently capture the dependency among the embedded segments. In Section 3.3, using DSW embedding and TSA layer, we construct hierarchical encoder-decoder (HED) utilize information at different scales for final forecasting.
3.1 DIMENSION-EGGMENT WISE EMBEDDING
To motivate our approach, we first analyze the embedding methods of the previous Transformer-based models for MTS forecasting (Zhou et al., 2021; Wu al., 2021a; Liu et al., 2021a; la; Zhou et al., 2022). As shown in Fig. I (b), existing methods embed data points at the same time step into a vector: X; > h X E D h, E Rdmmades where X₁ represents all the data points in D dimensions at step In this way, the input X1.7 is embedded into T vectors (h₃, he...., hr). Then the dependency among the vectors is captured for forecasting. Therefore, previous Transformer based models mainly capture cross-time dependency, while the cross-dimension dependency not explicitly captured during embedding, which limits their forecasting capability.
Transformer originally developed for NLP (Vaswani al., 2017), where each embedded vector represents an informative word. For MTS, a single value at step alone provides little information.
In this work, mainly focus on forecasting using only past data without covariates. But covariates can be easily incorporated in Crossformer through emhedding. Details shown in Appendix D.2.
3

---

Published as a conference paper at ICLR 2023
While it forms informative pattern with nearby values in time domain. Fig. 1 (a) shows typical attention score map of original Transformer for MTS forecasting. We can see that attention values have a tendency to segment, i.c. close data points have similar attention weights.
Based on the above two points, we argue that an embedded vector should represent a series segment of single dimension (Fig. 1 (c)), rather than the values of all dimensions at single step (Fig. (b)). To this end, we propose Snnension-Segment Wise (DSW) embedding where the points in each dimension are divided into segments of length Lseg and then embedded:
T Long
X1.T
sis
(1)
(8) Di - (1) x Leeg <t t S i x Lssey)
x 1.
where x E Lo is the i-th segment in dimension d with length Leeg. For convenience, we assume that T, 7 are divisible by Lsseg". Then each segment is embedded into vector using linear projection added with a position embedding:
+ h Ex!1) i,
E(Poe) i,
where E E Remodet Laco denotes the learnable projection matrix, and (pos) E Redmedel denotes the learnable position embedding for position (2, d). After embedding, obtain 2D vector array H = h, sis _T 1 S d d dD). where each h represents univariate time series segment. The idea segmentation also used in Du et al. (2022), which splits the embedded ID vector in order to enhance locality and reduce
sequence into segments to compute the computation complexity. However, like other Transformers for MTS forecasting, it does not explicitly capture cross-dimension dependency.
3.2 TWO-STAGE ATTENTION LAYER For the obtained 2D array H. one can flatten into ID sequence so that can be input to a canonical Transformer like ViT (Dosovitskiy et al., 2021) 1) does in vision. While we have specific considerations: 1) Different from images where the axes height and width are interchangeable. the axes time and dimension for MTS have different meanings and thus should be treated differently. 2) Directly applying self-attention on 2D array will cause the complexity of O(D2 r²)), which is unaffordable for large D. Therefore, we propose the Two-Stage Attention (TSA) Layer to capture cross-time and cross-dimension dependency among the 2D vector array, as sketched in Fig. 2 (a).
Given 2D array Z E BRLxDddodd as the input of the TSA Layer, where L and D are the number of segments and dimensions, respectively. Z here can be the output of DSW embedding or lower TSA layers. For convenience, the following, we use Ze denote the vectors of all dimensions at time step i, Z.d for those of all time steps in dimension d. In the cross-time stage, we directly apply multi-hean self-attention (MSA) to each dimension:
Cross-Time Stage
time LayerNorm Znd + MSA time (Z.di Zi,ds Zi,a)
MSAtimm
Z time_ LayerNorm
time + MLP (2 (time)
where 1 d D and LayerNorm denotes layer normalization as widely adopted in Vaswani et al. (2017); Dosovitskiy et al. (2021); Zhou et al. (2021), MLP denotes multi-layer (two in this paper) feedforward network, (SA K, V) denotes the multi-head self-attention (Vaswani et al., 2017) layer where Q K, V serve queries, keys and values. All dimensions (1 S d D) share the same MSA layer. time Ztime denotes the output of the MSA and MLP.
The computation complexity of cross-time stage is O(DI?). After this stage, the dependency among time segments in the same dimension is captured in time, Then time becomes the input of Cross-Dimensios Stage to capture gross-dimension dependency.
'If not, we pad them to proper length. See details in Appendix D.I.
4

---

Published as a conference paper at ICLR 2023
Z
Cross Time Steto
USS fur gins (
[
Cross Dimeesion Stage
Figure 2: The TSA layer. (a) Two-Stage Attention Layer to process 2D vector array representing multivariate time series: each vector refers to a segment of the original series. The whole vector array goes through the Cross-Time Stage and Pross-Dimension Stage to get corresponding dependency. (b) Directly using MSA in complexity. (c) Router mechanism for Pross-DDimensin Stage: small fixed number (c) of "routers" gather information from all dimensions and then distribute the gathered information. The complexity is reduced to O(2cD) = ODD).
Stage to build the D-to-I> D connection results in (DD)
Pross-Dimension Stage We can use large Long for long sequence in DSW Embedding to reduce the number of segments in cross- cross-time stage. While Cross Dosss--Dmnniinn Stage, we can not partition dimensions and directly apply MSA will cause the complexity of O (as shown in Fig. 2 (b)), which unaffordable for datasets with large D. Instead, we propose the router mechanism for potentially large D. As shown in Fig. (c), vectors for each time step as routers. These routers first aggregate messages from all dimensions by using routers as query in MSA and vectors of all dimensions key and value. Then routers distribute the received messages among dimensions by using vectors of dimensions as query and aggregated messages as key and value. In this way. the all-to-all connection among D dimensions are built:
small fixed number (c << D) of learnable
B MSA dim RRi, Zhime Z (ime)
1 Sish L
Z
dim
MSA2
dim time B.... Bi.),
LayerNorm time + Zz" dim)
dim LayerNorm
Z + *MLP(2 didm
where R E LExeddmoi (c con- stant) is the learnable vector array serving as routers. B E [xeedmodi the aggregated messages from all dimensions. Z dim denotes output of the router mechanism. All time steps (1 S < L) share the same MSA dim .MSAA dim. Zdi Zdim de- note output skip connection and MLP respectively. The router mech- anism reduce the complexity from O(D*L) to O(L)).
Froodde laser hojer
Hillen werterr
2002
20002
20003
zenc
UNW lanbestig
grnco
Adding up Eq. and Eq. 4, we model the stages as:
Figure 3: Architecture of the Hierarchical Encoder-Decode in Crossformer with 3 encoder layers. The length of each vector denotes the covered time range. The encoder (left) uses TSA layer and segment merging to capture dependency at different scales: a vector in upper layer covers longer range, resulting dependency at a coarser scale, Exploring different scales, the decoder (right) makes the final prediction by forecasting at each scale and adding them up.
Y Z (SA(Z)
where Z, E D Dx ddmated denotes the input and output vector array of TSA layer, respectively. Note that the overall computation complexity of the
5

---

Published as a conference paper at ICLR 2023
TSA layer is + = O(D After the Cross- and Cross-Dimension Stages, every two segments (i.e. Zi,d, dz) in Z are connected, as such both cross-time and cross dimension dependencies captured in Y.
3.3 HIERARCHICAL ENCODER -EEERDE
Hierarchical structures widely used in Transformers for MTS forecasting to capture information at different scales (Zhou et al., 2021; Liu et al., 2021a). In this section, we use the proposed DSW embedding, TSA layer and segment merging to construct Hierarchical Encoder-Decoder (HED). As shown in Fig. 3, the upper layer utilizes information at a coarser scale for forecasting, Forecasting values different scales are added output the final result.
Encoder In each layer of the encoder (except the first layer), every two adjacent vectors in time domain are merged to obtain the representation at coarser level, Then a TSA layer is applied to capture dependency at this scale. This process is modeled as Z Encoder( (Z (Znedd1))
1 1 1: en, = H
1> 1: a = M[Z Zand
sis 4-11 ISdED
(6)
Zenei TZA())
where H denotes the 2D array obtained by DSW embedding; z denotes the output of the 1-th encoder layer; M E Rddmode X2dmadet denotes a learnable matrix for segment merging: [-] denotes the concatenation operation; 1--1 denotes the number of segments in each dimension in layer 1 - 1, if it is not divisible by 2, we pad Zeme,/1 the proper length; Zened denotes the array after segment merging in the i-th layer. Suppose there are layers in the encoder, we use Zene;0 Zene, ..... N (Z (Zenc,) H) to represent the N + 1 outputs of the encoder. The
complexity of each encoder layer is O(D Deg
Decoder Obtaining the N 1 feature arrays output by the encoder, we N layers (indexed by 0, 1.....N) in decoder for forecasting. Layer takes the I-th encoded array as input, then outputs decoded 2D array of layer l. This process is summarized dee,, Decoder(Z dec,, Z Zene.
1 1 0: dec TSA TSA(E (((ec))
1>>0: 0: dec. TSA
Z. Z.doo doc
i,d MSA (Zlac Zene Zencl 15ddD
2 dec, LayerNorm ddeci dec,
Z dec, LayerNorm
ddc.. + MLP((Z
where (doc)) E R Tuue x DX DXdmodde denotes the learnable position embedding for decoder. dead is the output of TSA. The MSA layer takes Z as query and zenel as the key and value build the
connection between encoder and decoder. The output MSA is denoted as Z deo, Zdea,
denote the output of skip connection and MIP respectively. We use Zeec,0 Zene,1 dec, to represent the decoder output. The complexity of each decoder layer is (D=((e=))
Linear projection is applied each layer's output to yield the prediction of this layer. Layer predictions are summed to make the final prediction (for 0, O).
(5), for / = O.....N. x
W'z dec. i.d
pred, X
(s), (s)) IisSS T Loeg
pred
predd T41IT+
X1+1:
1=0
where W E Xghaegddmon is a learnable matrix to project vector to a time series segment. (e). iid Lio denotes the i-th segment in dimension d of the prediction. All the segments in layer

---

Published as a conference paper at ICLR 2023
Table I: MSE/MAE with different prediction lengths. Bold/underline indicates the best/second. Results of LSTMa, LSTnet, Transformer, Informer on the first 4 datasets are from Zhou et al. (2021).
Modele Metric
LSTMa
1.TTee
MIGGNN
Transfonner
Informer
Autofeemer
Pyrafcemer
FEDiommer
Crossfommer
MSF
MAF
MSF
MAF
MSR
MAR
MSF
MAF
MSF
MAF
MSF
MAR
MSF
MAF
MSF
MAF
MSF
MAF
24
0.650
0.624
1,203
0.901
0.336
0.393
0.620
0.577
0.577
0.549
0.439
0.440
0.493
0.507
0,318
0.384
0.305
0.367
48
0,720
0.675
1.456
0.950
0.386
0.429
0.692
0.671
0.685
0.625
0.429
0.642
0.554
0.544
0.342
0.396
0.352
0.394
168
1,212
1.214
0.474
0.947
0.797
0:931
0.493
0.781
0.675
0.449
0,410
0.441
336
1.424
2,655
1.369
0.643
1.128
0.509
0.912
0.747
0.456
0.461
720
1,560
1.322
2.143
1.380
0.916
0.750
1.241
0.917
1.215
0.59
0.537
0.993
0.792
0.521
0.515
0.519
0.524
24
0.621
1.988
1.170
0.260
0.324
0.306
0.371
1.323
0.309
0410
0.310
0.290
0.364
0.243
WTI 677ml
48
1.392
0.930
1,990
1.215
0.385
0.488
0.465
0.470
0.494
0.503
0485
0.664
0.465
0.343
0.382
96
1.339
0.913
2.762
1.542
0,428
0.446
0
0.612
0.675
0.614
0.502
0.576
0.520
0.504
0.256
0.112
0.530
0.373
288
1.740
1.124
1.257
2,076
0,460
01488
1.162
0.879
1,056
0.7%6
0.004
0.522
0.729
0.657
0.39%
0.415
0.404
0.427
672
2,736
1,555
1.917
2.941
0.620
0.571
1:231
1.103
1.192
0.926
0.607
0.530
0.980
0.678
0.455
0.464
0.569
0,528
24
0,546
0.570
0.615
0.545
0.307
0.356
0.349
0.397
0.335
0.381
0.363
0.396
0.301
0.359
0.357
0.112
0.294
0.343
48
0,829
0.677
11500
115889
0.888
0.422
0.386
0.433
1.395
04499
0.456
0.402
0.5%
0.421
0.425
0.458
0.370
0.411
168
1.038
0.835
0.748
0.647
0.498
0.512
0.63
0.582
0.608
0.567
0.574
0.548
0.11
0.531
0.564
0.541
0,473
0.494
336
1.657
1,059
0.783
0.683
0.506
0.525
0.707
0.634
0.702
0.620
0.600
0.571
0.539
0.543
0.533
0.336
0.695
0.515
1,536
0.851
0.757
0.527
0.741
0.587
0,696
0,369
0.645
01:33
0.241
0.478
0,229
0.326
0.156
0,574
0.676
0236
0.500
0.420
0.368
0/444
0.452
0.455
0,63
0.301
330
0.886
0.745
0419
0427
0220
0.23
0.499
0.381
0451
0.225
0.463
0.456
0,205
0.886
0.369
720
1.676
1895
0.56
0.565
0,42
0.10
0.391
0.438
D:406
0.43
0.372
0.480
0.461
0,372
0.423
960
1.591
1.128
0.605
0.599
0.471
0.451
0.192
0.550
0,450
0.588
0.366
0.426
0.550
0.489
0.391
0.489
0.433
0.458
24
4.220
1.385
4.075
1.600
4.265
1.387
3,054
1.323
4,588
1.452
3.101
1,238
3.970
1.338
2.687
1.147
3,041
1188
10.1
36
4,771
1.427
5.322
1,699
4.777
1.496
4.167
1.360
1.845
1,466
3.191
1,270
4.377
1.400
2.887
1.160
3,606
L252
48
4,945
.462
5.425
1,632
5.333
1.392
4.746
1.463
4.865
1,516
2.947
1.300
4.811
1,503
2.797
1.155
3.459
1.231
00
5.176
1.504
5.477
1,675
5.070
1.552
5.219
1.553
5.212
1,576
3,019
1.20
5.206
1.588
2,809
1.163
3.640
1.305
24
0.668
0.378
0.648
0.403
0.506
0.278
0.597
0.332
0.608
0.334
0.550
0.363
0.606
0.338
0.562
0.375
0
0.274
Trffie
48
0,709
0.400
0.709
0.425
0.29
0.658
0,369
0.644
0.595
0,376
0.619
0.346
0.567
0.374
0.519
0.295
168
0.435
0.19
0.363
0.660
0.649
0.635
0.347
0.289
0.747
0.624
0.64]
0.389
1.461
0.792
0.674
0.364
0.378
0.313
2 l are rearranged to get the layer prediction X++1 Predictions of all the layers are summed to obtain the final forecasting xree
x
4 EXPERIMENTS
4.1 PROTOCOLS
Datasets We conduct experiments on six real-world datasets following Zhou et al. (2021); Wu et al. (2021a). 1) ETTh1 (Electricity Transformer Temperature-hourlyy, 2) ETTml (Electricity Transformer Temperature-mmnntel]
3) WTH (Weather), 4) ECL (Electricity Consuming Load), 5) ILI (Influenza- Like Illness), 6) Traffic. The train/val/test splits for the first four datasets are same as Zhou et al. (2021), the last two are split by the ratio of 0.7:0.1:0.2 following Wu et al. (2021a).
Baselines We use the following popular models for MTS forecasting as baselines: LSTMa (Bah- danau et al., 2015), 2) LSTnet (Lai al., 2018). 3) MTGNN (Wu al., 2020), and recent Transformer-bases models for MTS forecasting: 4) Transformer (Vaswani et al., 2017), 5) In- former (Zhou et al., 2021), 6) Autoformer (Wu et al., 2021a), 7) Pyraformer (Liu et al., 2021a) and 8) FEDformer (Zhou et al., 2022).
Setup We use the same setting as in Zhou et al. (2021): train/val/test sets are zero-mean normalized with the mean and std of training set. On each dataset, we evaluate the performance over the changing future window size 7. For each т, the past window size T is regarded as a hyper-parameter to search which is common protocol in recent MTS transformer literature (Zhou al., 2021; Liu et al., 202 a)) We roll the whole set with stride- =1 1 to generate different input-output pairs. The Mean Square Error (MSE) and Mean Absolute Error (MAE) are used as evaluation metrics. All experiments are repeated for 5 times and the mean of the metrics reported. Our Crossformer only utilize the past series to forecast the future, while baseline models use additional covariates such as hour-of-the-day. Details about datasets, baselines, implementation, hyper-parameter are shown in Appendix A.
4.2 MAIN RESULTS
As shown in Table 1, Crossformer shows leading performance on most datasets, as well as on different prediction length settings, with the 36 top-1 and 51 top-2 cases out of 58 in total. It is worth noting that, perhaps due to the explicit use of cross-dimension dependency via GNN, MTGNN outperforms many braddd baselines. While MTGNN has been rarely compared in existing
7

---

Published as a conference paper at ICLR 2023
Table 2: Component ablation of Crossformer: DSW embedding, TSA layer and HED on ETThl.
Models
Transformer
DSW
DSW+TSA
DSW
DSW+TSA+HED
Metric
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
24
0.620
0.577
0.373
0.418
0.322
0.373
0.406
0.454
0.305
0.367
48
0.692
0.671
0.456
0.479
0.365
0.403
0.493
0.512
0.352
0.394
168
0.947
0.797
0,947
0.731
0.473
0,479
0.614
0.583
0.410
0.441
336
1.094
0.813
0.969
0.752
0.553
0.534
0.788
0.676
0.440
0.461
720
1.241
0.971
1.086
0.814
0.636
0.599
0.841
0.717
0.519
0.524
F-40
7-770
11:00
0.0
Creattoner
1 moroor 0CC wottonotoonn
mecehaaaconn 0Sei "Sk Dxcstimer
04
120
(a)
(b)
(c)
(d)
Figure Evaluation on hyper-parameter impact and computational efficiency. (a) MSE against hyper-
parameter segment length Teseg in DSW embedding on ETTh1. (b) MSE against hyper-parament
number of routers e the Cross-Dimension Stage of TSA layer on ETTh1. (c) Memory occupation
against the input length T on ETTh1. (d) Memory occupation against number of dimensions D on synthetic datasets with different number of dimensions.
transformers for MTS forecasting literatures. FEDformer and Autoformer outperform our model on ILI. We conjecture this because the size of dataset ILI is small and these two models introduce the prior knowledge of sequence decomposition into the network structure which makes them perform well when the data limited. Crossformer still outperforms other baselines on this dataset.
4.3 ABLATION STUDY
In our approach, there are three components: DSW embedding, TSA layer and HED. We perform ablation study on the ETTh dataset in line with Zhou al. (2021); Liu al. (2021a). We use Transformer as the baseline and DSW+TSA+HRS to denote Crossformer without ablation. Three ablation versions are compared: 1) DSW 2) DSW+TSA 3) DSW+HED.
We analyze the results shown in Table 2. 1) DSW performs better than Transformer on most settings. The only difference between DSW and Transformer is the embedding method, which indicates the usefulness of DSW embedding and the importance of cross-dae diose-dimnnioon dependency. 2) TSA constantly improves the forecasting accuracy. This suggests that it is reasonable to treat time and dimension differently. Moreover, TSA makes it possible to use Crossformer on datasets where the number of dimensions is large (e.g. D = 862 for dataset Traffic). 3) Comparing DSW+HED with DSW, HED decreases the forecasting accuracy when prediction length is short but increases it for long term prediction. The possible reason is that information at different scales is helpful to long term prediction. 4) Combining DSW, TSA and HED, our Crossformer yields best results on all settings.
4.4 EFFECT OF HYPER-PARAMETER PARAMETERS
We evaluate the effect of two hyper-parameters: segment length (Leeg in Eq. 1) and number of routers in TSA (c in Cross- DimsesDnnnn Stage of TSA) on the ETTh dataset. Segment Length: In Fig, 4(a), we prolong the segment length from 4 to 24 and evaluate MSE with different prediction windows. For short-term forecasting (T 24, 48), smaller segment yields relevantly better results, but the prediction accuracy is stable. For long-term forecasting (r 2 168), prolonging the segment length from to 24 causes the MSE to decrease. This indicates that long segments should be used for long-term forecasting. We further prolong the segment length to 48 for 336, 720, the MSE is slightly larger than that of 24. The possible reason is that 24 hours exactly matches the daily period of this dataset, while 48 is too coarse to capture fine- gnainndd information. Number of Routers in TSA Layer: Number of Routers e controls the information bandwidth among all dimensions. As Fig. 4(b) shows, the performance of Crossformer is stable w.r. to e for - < 336. For 720, the MSE is
8

---

Published as a conference paper at ICLR 2023
large when 3 but decreases and stabilizes when 2 5. In pratice, set 10 to balance the prediction accuracy and computation efficiency.
4.5 COMPUTATIONAL EFFICIENCY ANALYSIS
The theoretical complexity per layer of Transformer-basse models is compared in Table 3. The complexity of Crossformer encoder is quadratic w.r.t T' However, for long-term prediction where large Lee is used, the coefficient Tee. term can signif-
Table 3: Computation complexity per layer of Transformer-based models. T denotes the length of past series, T denotes the length of prediction window, D denotes the number of dimensions, Ley denotes the segment length of DSW embedding in Crossformer.
Method
Facoder layer
Decader layer
icantly reduce its practical complexity. We evaluate the memory occupation of these models on ETTh1.* We set the predic- tion window т 336 and prolong input length T. For Crossformer, Leeg is 24, which is the best value for 2 168
Transformer (Vaswani al. 2017)
O125)
Oiriz+ T)) (rr(r I T))
Informer (Zhou et al., 2021)
OfT (og
Autofommer (Wu
2021s)
O(Tlog
on 1l +r)) O(rir 2))
Pyraformer (Liu et al., 2021a)
O(T)
HEDtormer (Zhou et al., 2022)
0
O ++)
Crossformer (Ours)
(see Fig. 4 (a)). The result Fig. (c) shows that Crossformer achieves the best efficiency among the five methods within the tested length range. Theoretically, Informer, Autoformer and FEDformer are more efficient when T approaches infinity. In practice, Crossformer performs better when T is not extremely large (e.g. T S 10a).
We also evaluate the memory occupation w.r.t the number of dimensions D. For baseline models where cross-dimension dependency is not modeled explicitly, D has little effect. Therefore, we compare Crossformer with its ablation versions in Section 4.3. We also evaluate the TSA layers that directly use MSA in Router). Fig. (d) shows that Crossformer without TSA layer (DSW and DSW+HED) has quadratic complexity w.r.t D. TSA(w/o Router) helps reduce complexity and the Router mechanism further makes the complexity linear, so that Crossformer can process data with D = 300. Moreover, HED can slightly reduce the memory cost and we analyze this is because there are less vectors in upper layers after segment merging (see Fig. 3). Besides memory occupation, the actual running time evaluation shown in Appendix B.6.
Stage without the Router mechanism, denoted as TSA(w/o
5 CONCLUSIONS AND FUTURE WORK
We have proposed Crossformer, Transformer-baseed model utilizing cross-dimension dependency for multivariate time-series (MTS) forecasting. Specifically, the Dimension -eessinnn-egeen--- Wise (DSW) embedding embeds the input data into 2D vector array to preserve the information of both time and dimension. The Two-Stage- Stage-AAtention (TSA) layer is devised to capture the cross-tima and cross- dimension dependency of the embedded array. Using DSW embedding and TSA layer, Hierarchical Decoder (HED) is devised to utilize the information at different scales. Experimental results on six roal-world datasets show its effectiveness over previous tf
We analyzed the limitations of our work and briefly discuss some directions for future research: 1) In Cross-Dimension Stage, we build simple full connection among dimensions, which may introduce noise on high-ding sigh-dimensional datasets. Recent sparse and efficient Graph Transformers (Wu et al., 2022) can benefit our TSA layer on this problem. 2) A concurrent work (Zeng al.. 2023) which was accepted after the submission of this work received our attention. It questions the effectiveness of Transformers for MTS forecasting and proposes DLinear that outperforms all Transformers including our Crossformer on three of the six datasets (details are in Appendix B.2). It argues the main reason Therefore, enhancing the ordering preserving
is that MSA in Transformer is permutation-invariaa capability of Transformers is promising direction to overcome this shortcoming. 3) Considering datasets used in MTS analysis are much smaller and simpler than those used in vision and texts, besides new models, large datasets with various patterns also needed for future research.
*Pyraformer not evaluated it requires the additional compiler TVM achieve linear complexity.

---

Published as a conference paper at ICLR 2023
REFERENCES
Rafal A. Angryk, Petrus C. Martens, Berkay Aydin, Dustin J. Kempton, Sushant S. Mahajan, Sunitha Basodi, Azim Ahmadzadeh, Xumin Cai, Soukaina Filali Boubrahimi, Shah Muhammad Hamdi, Michael A. Schuh, and Manolis K. Georgoulis. Multivariate time series dataset for space weather data analytics. Scientific Data, 2020.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In International Conference Learning Representations (ICLR), 2015.
Defu Cao, Yujing Wang, Juanyong Duan, Ce Zhang, Xia Zhu, Congrui Huang, Yunhai Tong, Bixiong Xu, Jing Bai, Jie Tong, and Qi Zhang. Spectral temporal graph neural network for multivariate time-series forecasting. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Chun-Fu Chen, Quanfu Fan, and Rameswar Panda. Crossvit: Cross-attention multi-scale vision transformer for image classification. IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
Weiqi Chen, Wenwei Wang. Bingqing Peng, Qingsong Wen, Tian Zhou, and Liang Sun. Learning to rotate: Quaternion transformer for complicated periodical time series forecasting. In ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD), 2022.
Omer Fahrettin Demirel, Selim Zaim, Ahmet Caliskan, and Pinar Ozuyar. Forecasting natural gas consumption in Istanbul using neural networks and multivariate time series methods. Turkish Journal of Electrical Engineering and Computer Sciences, 2012.
Jacob Devlin, Ming-Wes Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT), 2019.
Linhao Dong, Shuang Xu, and Bo Xu. Speech-transforne no-recurrence sequence-to-sen to-uuueecee model for speech recognition. In IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR), 2021.
Dazhao Du, Bing Su, and Zhewei Wei. Preformer: Predictive transformer with multi-scale segment- wise correlations for long-term time series forecasting. arXiv preprint 22Xiv:2202.113366:: 2022.
Philipp Dufter, Martin Schmitt, and Hinrich Schütze. Position information in transformers: An
overview. Computational Linguistics, 2022.
Valentin Flunkert, David Salinas, and Jan Gasthaus. Deepar: Probabilistic forecasting with autore- gressive recurrent networks. International Journal of Forecasting, 2017.
Jake Grigsby, Zhe Wang, and Yanjun Qi. Long-range transformers for dynamic spatiotemporal forecasting, arXiv preprint
2022.
Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. Transformer in transformer. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Kai Han, Yunhe Wang. Hanting Chen, Xinghao Chen, Jianyuan Guo. Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, et al. A survey on vision transformer. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2022.
Guolin Ke, Di He, and Tie-Yan Liu. Rethinking positional encoding in language pre-training. In International Conference on Learning Representations (ICLR), 2021.
Lutz Kilian and Helmut LÅtkepohl. Structural Vector Autoregressive Analysis. Cambridge University
Press, 2017.
10

---

Published as a conference paper at ICLR 2023
Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long- and short-term temporal patterns with deep neural networks. In International ACM SIGIR Conference on Research & Development in Information Retrieval (SIGIR), 2018.
Colin S. Lea, Michael D. Flynn, René Vidal, Austin Reiter, and Gregory Hager. Temporal convolu- tional networks for action segmentation and detection. In IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2017.
Longyuan Li, Junchi Yan, Xiaokang Yang, and Yaohui Jin. Learning interpretable deep state space model for probabilistic time series forecasting. In International Joint Conference on Artificial Intelligence (IJCAI), 2019a.
Longyuan Li, Jihai Zhang, Junchi Yan, Yaohui Jin, Yunhao Zhang, Yanjie Duan, and Guangjian Tian. Synergetic learning of heterogeneous temporal sequences for multi- horizon probabilistic forecasting. In AAAI Conference on Artificial Intelligence (AAAI), 2021.
Shiyang Li. Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xine Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. In Advances in Neural Information Processing Systems (NeurIPS), 2019b.
Yaguang Li, Rose Yu, Cyrus Shahabi, and Yan Liu. Diffusion convolutional recurrent neural network: Data-driven traffic forecasting. In Infernational Conference Learning Representations (ICLR), 2018.
Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li. Weiyao Lin, Alex X Liu, and Schahram Dust- dar. Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting. In International Conference Learning Representations (ICLR), 2021a.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In IEEE/CVF International Conference on Computer Vision (ICCV), 2021b.
Andrew Patton. Copula methods for forecasting multivariate time series. Handbook of economic forecasting, 2013.
Yao Qin, Dongjin Song, Haifeng Chen, Wei Cheng, Guofei Jiang, and G. Cottrell. A dual-stage attention- based recurrent neural network for time series prediction. In International Joint Confer- ence on Artificial Intelligence (IJCAI), 2017.
Syama Sundar Rangapuram, Matthias W. Seeger, Jan Gasthaus, Lorenzo Stella, Bernie Wang, and Tim Januschowski. Deep state space models for time series forecasting. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference Machine Learning (ICML), 2021.
Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transform- ers with coro-correlttion for long-term series forecasting. In Advances in Neural Information Processing Systems (NeurIPS), 2021a.
Kan Wu, Houwen Peng, Minghao Chen, Jianlong Fu, and Hongyang Chao. Rethinking and improving relative position encoding for vision transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021b.
Qitian Wu, Wentao Zhao, Zenan Li, David Wipf, and Junchi Yan. Nodeformer: A scalable graph struc- ture learning transformer for node classification. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
11

---

Published as a conference paper at ICLR 2023
Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang. Xiaojun Chang, and Chengqi Zhang. Connecting the dots: Multivariate time series forecasting with graph neural networks. In ACM SIGKDD International Conference Knowledge Discovery & Data Mining (KDD), 2020.
Ting Yu, Haoteng Yin, and Zhanxing Zhu. Spatio-temporal graph convolutional networks: A deep learning framework for traffic forecasting. In International Joint Conference on Artificial Intelligence (IJCAI), 2018.
Chulhee Yun, Srinadh Bhojanapalli, Ankit Singh Rawat, Sashank Reddi, and Sanjiv Kumar. Are trans- formers universal approximators of sequence-to-sequenee functions? In International Conference on Learning Representations (ICLR), 2020.
Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? In AAAI Conference on Artificial Intelligence (AAAI), 2023.
Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wan Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In AAAI Conference on Artificial Intelligence (AAAI), 2021.
Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. Fedformer: Frequency enhanced decomposed transformer for long-terme series forecasting. In International Conference on Machine Learning (ICML), 2022.
12

---

Published as a conference paper at ICLR 2023
A DETAILS OF EXPERIMENTS
A.1 BENCHMARKING DATASETS
We conduct experiments the following six real-world datasets following Zhou al. (2021); Wu et al. (2021a):
ETTh1 (Electricity Transformer Temperature-hourlyy) contains indicators of electricity transformer in two years, including oil temperature, useful load, etc. Data points are recorded every hour and train/val/test 12/4/4 months.
2) ETTml (Electricity Transformer cemperature-minutely contains the same indicators as ETTh1 but data points are recorded every miniutes. Train/val/test split is same as ETTh1.
3) WTH (Weather) contains 12 meteorological indicators in U.S. in 4 years, including visibility, wind speed, etc. Train/val/test is 28/10/10 months.
4) ECL (Electricity Consuming Load) contains hourly electricity consumption (in Kwh) of 321 clients in two years. Train/val/test is 15/3/4 months.
5) ILI (Influenza-Like Illness) contains 7 weekly recorded indicators of patients data from Centers for Disease Control and Prevention of the United States between between 2002 and 2021. The ratio of train/validation/testt splits is 0.7:0.1:0.2.
6) Traffic contains hourly road occupancy rates measured by 862 sensors on San Francisco Bay area freeways in 2 years. The ratio of rrain/validaiionneest splits 0.7:0.1:0.2.
The train/val/test splits for ETTh1, ETTmI, WTH, ECL are same as Zhou et al. (2021), for ILI and Traffic are same as Wu et al. (2021a).
The first four datasets are publicly available https://github com/ zhouhaoy1/ Informer2020 and the last two are publicly available at https: //gttttttuuu/// com/ httmmtul///// Autoformer.
A.2 BASELINE METHODS
We briefly describe the selected baselines:
1) LSTMa (Bahdanau et al., 2015) treats the input MTS as a sequence of multi-dimensional vectors.
It builds encoder-decoder using RNN and automatically aligns target future steps with their relevant past.
2) LSTnet (Lai al., 2018) uses CNN extract cross-dimension dependency and short cross- time dependency. The long-term coss-time dependency is captured through RNN. The source code available at https: / / q thub. com/ waiiuuuuunnSSTNe
3) MTGNN (Wu et al., 2020) explicitly utilizes cross-dimension dependency using GNN. A graph learning layer learns a graph structure where each node represents one dimension in MTS. Then graph convolution modules are interleaved with temporal convolution modules to explicitly cap- ture cross-dimension and cross- dependency respectively. The source code is available at https::/// httptbitttubbb com/ htt//nzhann GNG.N.
4) Transformer is closed to the original Transformer (Vaswani et al., 2017) that uses self-attention mechanism capture cross-time dependency. The Informer-style one-step generative decoder is used for forecasting, therefore this is denoted as Informer in Informer (Zhou et al., 2021).
5) Informer (Zhou al., 2021) is Transformer-basee model using the ProbSparse self-attention to capture cross-time dependency for forecasting. The source code of Transformer and Informer is available at https: / httpt/gtttbt/bt// com/ ztw.haoyiic//o/ooo /tti//ffrmnn 2020..
model using decomposition architecture with CorrelCCotlltoon mechanism to capture cross-time dependency for forecasting. The source code is available at https: / / thub.. com/
6) Autoformer (Wu et al., 2021a) is a Transformer-
coomthunm t/uml/humm////mmmmm Auttoff rmer.
7) Pyraformer (Liu et al., 2021a) is Transformer-basee model learning multi-resolution represen- tation of the time series by the pyramidal attention module to capture cross-time dependency for forecasting. The source code available at https: / thub. com/ a
r. model that uses the reasonal-trend de-
8) FEDformer (Zhou et al., 2022) is Transformer-bastt composition with frequency enhanced blocks to capture cross-time dependency for forecasting. The source code is available at https: //gi thub. com/ MMn//MAzii /g/FFD
13

---

Published as a conference paper at ICLR 2023
A.3
SELECTION AND IMPLEMENTATION DETAILS
A.3.1 MAIN EXPERIMENTS
For the main experiments, we use the Crossformer with 3 encoder layers. The number of routers in TSA layer c is set to 10. For dataset ETTh1, ETTm1, WTH and ILI, dimension hidden state dmadel is dimension of hidden state moddel is set to 64, the head number of multi-head attention is set to 2. The segment length Leg is chosen from {6, 12, 24] via grid search. We use MSE as loss function and batch size is set to 32. Adam optimizer is used for training and the initial learning rate is chosen from (3e-3, le-3, 5e-4, le-4, 5e-5, le-5) via grid search. The total number of epochs is 20, the validation loss does not decreases within three epochs, the training process will stop early.
256, the head number of multi-head attention is
4; For dataset ECL and Traffic,
For baseline models, if the original papers conduct experiments on the dataset we use, the hyper- parameters (except input length 7) recommended in the original papers are used, including the number of layers, dimension hidden states, etc. Otherwise, the hyper-parameters are chosen through grid search using the validation set.
Following Zhou et al. (2021), on datasets ETThI, WTH, ECL and Traffic, for different prediction length the input length T is chosen from {24, 48, 96, 168, 336, 720]; on ETTml, the input length chosen from {24, 48, 96, 192, 288, 672]: on ILI, the input length is chosen from {24, 36, 48, 60).
All models including Crossformer and baselines are implemented in PyTorch and trained on single NVIDIA Quadro RTX 8000 GPU with 48GB memory.
A.3.2 EFFICIENCY ANALYSIS
To evaluate the computational efficiency w.r.t the input length T in Figure 4(c) of the main paper, we align the hyper-parameters of all Transformer- based models as follows: prediction length 7 to 336, number of encoder layers is set to 2, dimension of hidden state dmodet is set to 256, the head number of multi-head attention set to 4.
To evaluate the computational efficiency w.r.t the number of dimensions D in Figure 4(d) of the main paper, we align the perea-parameters of ablation versions of Crossformer as follows as: both input length T' and prediction length are set to 336, number of encoder layers is set to 3, dmodel is set to 64, the head number of multi-head attention is set to
Experiments in the computational efficiency analysis section are conducted on single NVIDIA GeForce RTX 2080Ti GPU with 1GB memory.
A.4 DETAILS OF ABLATION VERSIONS OF CROSSFORMER
We describe the models we used in ablation study below:
DSW represents Crossformer without TSA and HED. The input is embedded by DSW embedding and flatten into ID sequence to be input to the original Transformer. The only difference between this model and the Transformer is the embedding method.
2) DSW+TSA represents Crossformer without HED. Compared with Crossformer, the encoder does not use segment merging to capture dependency at different scales. The decoder takes the final output of encoder (i.c. Z N) as input instead of using encoder's 's output at each scale.
3) DSW+HED represents Crossformer without TSA. In each encoder layer and decoder layer, the 2D vector array is flatten into a ID sequence to be input to the original self-attention layer for dependency capture.
B EXTRA EXPERIMENTAL RESULTS
B. SHOWCASES OF MAIN RESULTS
Figure 5 shows the forecasting cases three dimensions of the ETTml dataset with prediction length 288. For dimension "HUFL"" all the five models capture the periodic pattern, but Crossformer is
14

---

Published as a conference paper at ICLR 2023
MTGNN
Autoformer
Pyraformer
FEDformer
1
4
Crossformer
HUFL
HULL
(c) LUFL
Figure 5: Forecasting cases of three dimensions: High UseFul Load (HUFL), High UseLess Load (HULL) and Low UseFul Load (LUFL) of the ETTml dataset with prediction length 288. The red / blue curves stand for the ground truth / prediction. Each row represents one model and each column represents one dimension.
the closest to the ground truth. For "HULL", Pyraformer fails to capture the periodic pattern from the noisy data. For "LUFL" where the data has no clear periodic pattern, MTGNN, FEDformer and Crossformer capture its trend and show significantly better results than the other two models.
Figure shows the forecasting cases of three dimensions of the WTH dataset with prediction length T = 336. For dimension "DBT", all the five models capture the periodic pattern. For "DPT", Autoformer and FEDformer fails capture increasing trend the data. For "WD", all models capture the periodic pattern from the noisy data, and the cruves output by MTGNN and Crossformer are sharper than the other three models.
B.2 COMPARISON WITH EXTRA METHIODS
We further compare with two additional concurrent methods which were either not peer- reviewed (Grigsby al., 2022) or were accepted after the submission this work (Zeng al., 2023): 1) STformer (Grigsby et al., 2022), a Transformer-basee model that directly flattens the multivariate time-series XI.7 E RTDD into a ID sequence to be input to Transformers: 2) DLin- (Zeng et al., 2023), a simple linear model with seasonal-trent decomposition that challenges Transformer- based models for MTS forecasting. Results are shown in Table and LSTMa and LSTnet omitted they are not competitive with other models.
The basic idea of STformer is similar to our Crossformer: both of them extend the 1-D attention to dependency makes STformer competitive with previous
D. The explicit utilization of cross-dimeneen Transformer based models on ETThI, ETTml and WTH. especially for short-term prediction. However, STformer directly flattens the 2-D time series into 1-D sequence be input the
15

---

Published as a conference paper at ICLR 2023
MTGNN
Autoformer
Pyraformer
Crossformer FEDformer
DBT
(b) DPT
(c) WD
Figure 6: Forecasting cases of three dimensions: Dry Bulb Temperature (DBT), Dew Point Tempera- ture (DPT) and Wind Direction (WD) of the WTH dataset with prediction length 7 = 336. The red blue curves stand for the ground truth / prediction. Each row represents one model and each column represents one dimension.
Transformer. This straightforward method does not distinguish the time and dimension axes and computationally inefficient. Therefore, besides the good performance for short-term prediction, STformer has difficulty long-term prediction and encounters the out-of-memory (OOM) problem on nigh-dimensional datasets (ECL and Traffic). While Crossformer uses the DSW embedding to capture local dependency and reduce the complexity. The TSA layer with the router mechanism is devised deal with the heterogeneity of time and dimension axis and further improve efficiency.
DLinear is on par with our Crossformer on ETThl and ETTml I (T < 96); has similar performance with FEDformer on ILI; performs worse than Crossformer on WTH; outperforms all Transformer- based models including our Crossformer on ETTml (r 2 288), ECL and Traffic, Considering its simplicity, the performance is impressive. Based on the results, we analyze the limitations of Crossformer and propose some directions to improve it in the future:
Stage of TSA layer, we simply build an all-to-all connection among D dimensions with the router mechanism. Besides capturing the cross-dimension dependency, this full connection also introduces noise, especially for high-dimensional dataset. We think high-dimensional data has the sparse property: each dimension is only relevant to small fraction of all dimensions. Therefore, utilizing the sparsity to reduce noise and improve the computation efficiency the TSA layer could be promising direction.
In
2) Authors of DLinear (Zeng et al., 2023) argue that the Transformer- based models have difficulty in preserving ordering information because the attention mechanism is cermutationiinaaria and the absolute position embedding injected into the model is not enough for time series forecasting, which an order-sensitive task. Although Yun al. (2020) theoretically proves that Transformers with
16

---

Published as a conference paper at ICLR 2023
Table 4: MSE/MAE comparison with extra methods: STformer (Grigsby et al., 2022) and DLin- ear (Zeng et al., 2023). Bold/underline indicates the best/second. OOM indicates out-of- out-of-memory model; yellow marks Transformer-basee
problem. Gray background marks the models where cross cross-dimension dependency is omitted; blue marks Transformer-basee models explic- itly utilizing cross-dimension dependency; red marks the linear model with series decomposition.
Models
MITGNN
Transformer
Informer
Ausoforner
Pyraformer
FEDfomer
STlurner
Croedormer
DLinear
Metre
MSE
MAE
MSE
MAE
MSB
MAE
MSE
MAE
MSE
MAE
MSB
MAE
MSE
MAE
MSE
MAF
MSE
MAE
24
0.336
0.395
0.620
0.577
0.577
(1549
0.429
0.441)
0.493
0.507
0.318
0.384
0.368
0.441
0.305
0.367
0.312
0.385
48
0,386
0.429
0.692
0.871
0.685
0.625
0.429
0.442
0.554
0.564
0.342
0.396
0.465
0.465
0.352
0.394
0.352
0.383
108
02.400
0.474
0.947
0.799
0.931
0.752
0.493
0.479
0.781
0.675
0.412
0.439
0.652
0.008
0.410
0.441
0,416
0.450
336
0.756
0.643
1.004
0.813
1.128
0.573
0.509
0.492
0.912
0.747
0.456
0.474
1.669
0.306
0.440
0.46
0.450
0.452
720
0,916
0.750
1.241
0.917
0.896
0.539
0.537
0.993
0.792
0.521
0.515
1.071
0.817
0.519
0.524
0,456
0,501
24
0.290
0.324
0.306
0.371
0.325
0.369
0.410
0.428
0.310
0.371
0.290
0.364
0.278
0.348
0.211
0.283
0.289
WTH E7Tml
48
0.386
0.465
0.470
0.464
0.465
0.342
0.445
0.350
0.428
0.681
0.612
0.64
0.502
0.520
0.366
0.420
0.455
0.320
0.354
280
0,009
0.488
1,162
0.879
14356
0.786
0.604
0.522
0:722
0.57
0.294
0,433
0.235
0.597
0,804
0.386
672
0.630
1:231
1.103
1.192
0.976
0.607
0.530
0.980
0.678
0.455
0
0.777
0.625
0.528
0.416
0.417
24
0.307
0.356
0.449
0.397
0.335
0.381
0.363
0.396
0.301
0,350
0.557
0,42
0.207
0.359
0.294
0.343
0.357
0.391
48
0.353
0.122
0.386
0/433
0,395
0.459
0.455
0.162
0,336
0,021
0.128
0,458
0,811
0.110
0,370
0.411
0.425
0.864
168
0.495
0.512
16613
0.582
0.608
11567
0.574
0.548
0.539
0.521
0156-
0.54
0.502
0,473
0.494
0.515
0.516
336
0.505
0.523
0.007
0.634
0.702
0.620
0.600
0.571
0.539
0.543
0.533
0.536
0566
0.564
0.495
0.515
0.536
0.557
720
0.510
0.527
0.834
0.741
0X831
0.731
0.587
0.570
0.547
0,553
0.562
0.557
0.589
0.582
0,526
0.542
0.582
0,571
48
0.173
0.280
1.334
1.399
0.344
0.393
0.241
0.251
11.778
0.471
0.229
0.33M
0.356
0.422
0.15%
0.255
0.155
0.258
168
0.256
0.120
0.353
0.420
0.368
0.424
0.290
0.387
0.452
0.455
0.561
0.361
0.516
0 0.537
0.31
0.909
0.195
0.287
EC1
336
0,528
0.373
0.381
0.639
0
0.431
0.375
0.426
0.463
0.456
0.105
0.386
00M
0.369
0,238
0.316
0.422
0.410
0
0.438
01443
0,480
0,434
OOM
0.425
0.346
0.471
0.550
0.548
0.366
0.550
0.593
00M
0.438
0.367
4,265
1.323
1:462
1,238
3,970
2,887
3.150
3,041
11I1
36
4.777
1.496
4.167
1.300
1.496
3.3M7
1.27U
4.377
L.10
2,89
1,160
3.122
1.243
3,406
L184
48
5.333
1.592
4.746
4.865
1.516
2.947
4.811
1.503
2.77
1,155
3,499
1.155
60
5,070
1,552
5.219
1,553
5.212
1.576
3.019
1.202
5,204
1,588
2,809
1,163
3.715
1.316
3,640
1.305
3,011
1,245
24
0.505
0.278
0.591
0,332
0.608
0.334
0,550
0.363
0,506
0,338
0.562
0,375
0.747
0.447
0,091
0.274
0.351
0.261
Traffic
48
0.512
0.298
0.658
1.360
0.644
0.359
0.595
0.376
0.619
0.346
(1567
0.777
ODM
0.19
0.905
0.370
0.270
168
0,521
0.319
0.664
0.363
0.660
0.391
0.649
0.407
0.635
0.347
0.607
0.365
00M
0.513
0.789
0.395
0.277
336
0,540
0.335
0.684
0.358
0.747
0.405
0.624
0.388
0.641
0.347
0.624
0.389
00M
0.330
0.300
0.415
0.289
720
0.557
0.343
0.685
0.370
0.792
0.400
0.674
0.417
0.670
0,3 0.364
0.63
0.378
00M
0.513
0,313
0.455
0.313
trainable positional embedding are universal approximators of lequeence-t-ssqueeee functions, the ordering information still needs to be enhanced in practice. We think that relative position encoding in texts (Ke et al., 2021; Dufter et al., 2022) and vision (Wu et al., 2021b) could be useful for ordering information enhancement.
3) The sizes of datasets used for time series forecasting are much smaller than those for texts and vision, and the patterns in time series datasets are also simpler. Considering vision transformers surpass inductive bias and achieves excellent results compared CNNs after pre-trained on large amounts of data (Dosovitskiy et al., 2021), Transformers for time series may also require large-size datasets with various patterns exploit their full potential.
As quoted from the paper, authors mentioned that DL.inear "does not model correlations among variates". Therefore, incorporating cross-dimension dependency into DLinear further improve prediction accuracy is also a promising direction. Moreover, our DSW embedding to enhance locality and HED to capture dependency at different scales can also be potentially useful to further inspire and enhance DLinear.
B.3 ABLATION STUDY OF THE ROUTER MECHANISM
The ablation study of the three main components of Crossformer is shown in See. 4.3. In this section, we conduct an ablation study of the router mechanism, a sub-module in TSA layer, and evaluate its impact on prediction accuracy. should be noticed that the router mechanism is mainly proposed to reduce the computation complexity when D is large. Results are shown in Table 5. Adding TSA(W/o Router) constantly improves the prediction accuracy of DSW and DSW+HED, showing the necessity of capturing cross-time and cross-diments dinension dependency in two different stages. For short term prediction (7 < 168), the performances of TSA(w/o Router) and TSA are similar, no matter whether HED is used or not. For long term prediction (T 336), the router mechanism slightly improves the prediction accuracy. The possible reason is that we set separate routers for each time step, which helps capture long-term dependency that varies over time.
17

---

Published as a conference paper at ICLR 2023
Table Complementary results ablation study in Table 2. TSA(w/o Router) denotes TSA layer without the router mechanism that directly uses MSA in the Cross-Dimension Stage.
Models
DSW
DSW+
DSW+TSA
DIW+IIED
DSW+HEED+ TSA(W/o Router)
DSW+TSA-HED
TSA(Wlo Router)
Metric
MAB
MSE
MAE
MAE
MAE
MAE
0.373
0.418
0.320
0.376
0.373
0.406
0.454
0.311
0.375
0.367
0.456
0.479
0.356
0.396
0.365
0.403
0.493
0.512
0.363
0.406
0.352
0.394
168
0.947
0.731
0.487
0.493
0.473
0.479
0.614
0.583
0.416
0.444
0.410
0.441
336
0.969
0.752
0.585
0,564
0.553
0.534
0.788
0.676
0.487
0.499
0.440
0.461
720
1.086
0.814
0.665
0.615
0.636
0.599
0.841
0.717
0.540
0.542
0.519
0.524
Query Dim #1
Query Dim
Query Dim #5
(a) Query Time Segment T + T + (b) Query Time Segment T 19 T 21
Figure 7: Attention scores calculated by the decoder of ablation version of Crossformer (i.e. DSW) dataset ETTh1. The input length, prediction length and segment length set as T 168, = 24, Laeg = The x axis each figufigure represents the time steps serve as keys in attention mechanism, while the y axis denotes dimensions. Brighter color denotes higher attention weights.
B.4 DEPENDENCY VISUALIZATION
As the attention scores computed by Crossformer are abstract and hard to visualize, we visualize scores computed by the ablation version, DSW, in Figure In addition cross-time dependency that other Transformer models can compute, Crossformer also provides information about cross- dimension dependency. As shown Figure when predicting Dim #1, the model focus on both Dim #1 and #3. When predicting Dim #5, instead of focus on Dim #5 itself, more attention is paid to Dim #4.
B.5 HIERARCHICAL PREDICTION PATTERN VISUALIZATION
Figure 8 shows the hierarchical prediction patterns output by our HED. The top prediction layer, Layer 3, captures the low frequency general trend and periodic pattern of the future value. By adding predictions at finer scales, finer high frequency patterns are added and the prediction get closer to the ground truth curve.
B.6 RUNNING TIME EFFICIENCY ANALYSIS
In the main paper, we show the memory occupation w.r.t input length T and number dimensions D. Here evaluate the running time. Figure (a) shows the running time per batch of Crossformer
18

---

Published as a conference paper at ICLR 2023
1.0
0.5
103
1.0
Truth
(a) Layer Prediction
(b) Layer 2 ~ 3 Prediction
00-
100
300
100
359
(c) Layer 3 Prediction
(d) Layer ~ 3 Prediction
Figure 8: Hierarchical prediction visualization of ETTml with dimension HUFL and prediction
length 7 = 288. From top left to bottom right, we gradually add layer predictions at finer scales.
Translsmer lomer
DSS
Computation (sibatch)
1.0
Informer
Autoformer
Computation Time (s/batch)
DSW++ED
DSW++15A
ISW--TSAES Reafer)-H81
6.8
0.6
0.1
0.0
(a)
(b)
Figure 9: Evaluation on computational speed. (a) Running time per batch w.r.t the input length 7 on
ETTh1. (b) Running time per batch w.r.t number of dimensions D on synthetic datasets by different numbers of dimensions.
and other Transformer based models w.r.t input length T. FEDformer is much slower than other Transformer based models. Crossformer achieves the best computation speed among the five methods within the tested length range.
Figure 9 (b) shows the running time per batch of Crossformer and its ablation versions w.r.t the number of dimensions D. Crossformers without TSA layer (DSW and DSW+HED) are faster when D is small (D < 30). However, they have difficulty processing nigh-dimensional MTS due to the quadratic complexity w.r.t D. Indeed, for single NVIDIA Geforce RTX 2080Ti GPU with 11GB memory, DSW and DSW+HED encounters the out-of-ment mieeooymmmryy (OOM) problem when D > 50. Moreover, TSA(w/o Router) encounter the OOM problem when 200.
C DISCUSSION ON THE SELECTION OF HYPER- EYPER-PARAMMTERS
We recommend to first determine the segment length Luege as it is related to both the model perfor- mance and computation efficiency. The general idea is to use small Leeg for short-term prediction and large Leeg for long- torg-term prediction. Some priors about the data also help to select L For example, if the hourly sampled data has daily period, it better set Taeg 24. Next, select the number
19

---

Published as a conference paper at ICLR 2023
Table 6: MSE and MAE evaluation with different segment lengths on ETTml dataset. denotes
segment length used in the main text, which is divisor of T,
Metric
MSE
MAE
MSE
MAE
MSE
MAE
Segment Length Issg
5
6
7
T' 288, 48
0.291
0.349
0.300
0.352
0.284
0.346
Segment Length Laeg
22
24
26
T 672, T 288
0.401
0.424
0.404
0.427
0.409
0.429
of layers for encoder and decoder N. Crossformer with larger N can utilize information of more scales, but also requires more computing resources. The number routers in TSA layer c can be set 10 balance the prediction accuracy and computation efficiency. Finally, dimension of hidden states demode and head number of multi-head attention can be determined based on the available computing resources.
D SUPPLEMENTARY DESIGN TO CROSSFORMER
D.1 HANDLING INDIVISIBLE LENGTH
In the main paper, we assume that the input length and prediction length r are divisible by segment length Leeg- In this section, use padding mechanism to handle cases where the assumption is not satisfied.
If is not divisible by Lseg, we have (ki L) <T < ki Lisg for some kn We pad kn Laeg - T duplicated x1 in front of X1:7 to get Xi:T²
x1-T
X1, X X1, X1:T]
1 Tiscu -T
where [:] denotes the concatenation operation. X1,TT E Bkilocy xD can be input to the encoder of Crossformer.
If is not divisible by Iseg. we have (k2 1)Lsee <T< < hog for some lgg We set the learnable position embedding for decoder as E(dee) E agkixDxdmant and input it to the decoder to get an output in shape of R Then the first steps of the output is used as x XT+11T+7*
pred
We conduct experiment on ETTml dataset to evaluate the effect of indivisible length. Results in Table 6 show that with padding mechanism, indivisible length does not degrade model performance, for both short-term prediction and long-term prediction.
D.2 INCORPORATINS COVARIATES
In the main text, only use historical series X to forecast the future 1:4+1:++-- In this section, we try to incorporate covariates C1.7+7 into Crossformer. We use straightforward method: first embed the covariates into point-wise vectors (d), d2,,d2,.. d2,.... ........ dr like previous Transformer-based models do (Zhou et al., 2021; Wu et al., 202 Liu et al., 2021a). Then, merge the point-wise vectors into segment-wiss vectors using learnable linear combination. Finally, add the segment-wiss vectors to each dimension of the 2D vector array obtained by DSW embedding:
C₁ - de T
d
Σ
a e,d(i-1jxtan 1ixbogg isis
T Leeg
(10)
DKjŠtzng
T 1 15d5D
h 000 hi, + (
1 1si5
Lseg
where - denotes embedding method for point- wise covariates. aj, S S Leeg denotes learnable factors for linear combination. ((()) denotes the segment-wise covariate embedding. hçve denotes the
20

---

Published as a conference paper at ICLR 2023
Table 7: MSE and MAE evaluation of Crossformer without/with covariates on ETThl dataset.
Models
Crossformer
Crossformer+Cov
Metric
MSE
MAE
MSE
MAE
0.305
0.367
0.308
0.368
48
0.352
0.394
0.358
0.399
168
0.410
0.441
0.412
0.440
336
0.440
0.461
0.438
0,465
720
0.519
0.524
0.522
0.531
embedded vector with covariate information for the i-th segment in dimension where hi.d the embedded vector obtained from DSW embedding in the main text. The processing for the input of the decoder is similar, the segment wise covariate embedding added to the position embedding for decoder, i.e. E(deo)
We conduct experiments on ETThl dataset to evaluate the effect covariates. Hour-of-the-day. day- of-the-week. mot-o and day-of-the-year are used covariates. Results in Table 7 show that incorporating covariates does not improve the performance of Crossformer. The possible reason this straightforward embedding method does not cooperate well with Crossformer. Incorporating covariates into Crossformer to further improve prediction accuracy is still an open problem.
21


Reviews:
[{"review_id": "tLzgr8ORjVG", "reviewer": "Reviewer_SiSN", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents Crossformer, a Transformer-based model for multivariate time series (MTS) forecasting, especially focusing on cross-dimension dependency, which has not been widely investigated so far. They propose Dimension-Segment-Wise (DSW) embedding and Two-Stage Attention (TSA) layer to address this issue and further build a Hierarchical Encoder-Decoder (HED) to combine the information at different scales for final forecasting. ", "summary_of_the_review": "The paper is well-written overall. I think the contribution and novelty of the proposed work are clearly shown. ", "strength_and_weaknesses": "[+] The idea for utilizing cross-dimension attention in multivariate time-series data seems novel and interesting. The reduced time complexity is also an advantage. \n\n[+] Its effectiveness is validated through extensive experiments across six datasets. Sensitivity analysis on hyperparameter setting and ablation study is also provided. \n\n[-] In computational efficiency analysis (Section 4.5), it would be better if the authors can also provide a comparison of actual execution time.\n\n[-] In the ablation study (Section 4.3), the MSE of the TSA layer with and without the routers should be compared to validate the proposed router mechanism. \n\n[-] The authors mention that HED decreases the performance when the prediction length is short, and the possible reason is the information on different scales is helpful to long-term prediction. However, when it is combined with TSA, short-length prediction shows good performance. The explanation of how TSA overcomes the shortcoming of HED should be addressed. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "pyYnTyd2NP", "reviewer": "Reviewer_pZRS", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This study addresses Multivariate time series forecasting using a transformer-based model. Namely, the authors presented Crossformer, a transformer based model which not only takes into account the temporal sequence, but also the cross dimension dependencies among the variables.\nThe inputs for Crossformer, is embedded using Dimension-Segment-Wise (DSW) embedding in a 2d array then it is processed by using a two-stage attention layer that  supposedly captures both cross-time and cross-dimension dependencies. \nIn the cross-time stage,  a multi-head self-attention (MSA) is applied to each dimension and then an MLP is used to capture the dependency among the different segments in each dimension. \nIn the cross-dimension stage, as directly applying MSA for high dimensional datasets will not be affordable, the authors proposed using router mechanism to aggregate messages from all dimensions into a low dimension space.\nThen to make the all-to-all connections,  the aggregated\nmessages are considered as key and value and the vectors of dimensions as query.\nFinally, the authors used a hierarchical  encoder-decoder structure to use the extracted information in different scales. \n", "summary_of_the_review": "In my opinion, this study is nicely done and by adding two mentioned comparison above will be completed enough to be published in this conference. ", "strength_and_weaknesses": "This manuscript is well written and organized very well. Also  the studied topic is very interesting as it is usually the case that for MTS the cross dimension dependencies is overlooked. \n\nThe authors argue that as “For MTS, a single value at a step alone provides little information”, embedding in a way that “nearby points over time form a segment” could be beneficial. This argument seems interesting and valid as it is used and illustrated nicely (In Figure 1) by the authors.\n\nIn my opinion this is an interesting study.   I think you can complete your great work by comparing your method with “Long-Range Transformers for Dynamic Spatiotemporal Forecasting” which you can find in “https://arxiv.org/pdf/2109.12218.pdf” and also include the very simple baseline that is mentioned in this study “https://arxiv.org/pdf/2205.13504.pdf”. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "l_jBBou99Sg", "reviewer": "Reviewer_74U5", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed `CrossFormer`, a Transformer based neural network for multivariate time series (MTS) forecasting. The author(s) claimed that cross-dimension dependency was not well utilized and developed a method to explicitly explore and utilize cross-dimension dependency for MTS forecasting. In order to achieve that, the authors employed `Dimension-Segment-Wise` embedding to aggregate segment based embedding on the time axis, and designed a two stage attention layer using multi-head self attention and MLP to capture cross-time and cross-dimension dependency of the embedded input. A hierarchical encoder-decoder is further introduced to utilize the information at different scales. In this way, long term dependency between different time steps and dimensions can be captured. The author(s) compared the accuracy of the method on six real-world datasets and demonstrated superior results in most of the tasks. They also analyzed the theoretical time complexity of the method to be $O(\\frac{D}{L_{seg}^2}{T^2})$ for encoder, $O(\\frac{D}{L_{seg}^2}{(\\tau(\\tau+T))})$ for decoder, which performed better in terms of memory occupation and computation time in empirical experiments.\n", "summary_of_the_review": "\nTo summarize, this paper was written in high quality. It proposed `CrossFormer`, a Transformer based neural network, to capture cross-time and cross-dimension dependency, and achieved state-of-the-art performance in most of the multivariate time series forecasting tasks benchmarked, especially in long time series. The experiments were extensive and covered aspects of accuracy, memory occupation, computation time. The ablation study was comprehensive and proved the effectiveness of architectures proposed and hyper-parameters selected. There's a bit of a lack of novelty and this method may be hard to extend to other areas of representation learning. However, the insights from cross-dimension dependency and new design of 2-stage attention layers are very important. Therefore, I still recommend this paper to be accepted.", "strength_and_weaknesses": "This paper is well organized with comprehensive introduction into the network details and extensive experiment results. The setting of multivariate time series forecasting was well-introduced. The methodology part clearly presented how the initial input was segmented and fed into the 2-stage attention layer to capture cross-time and cross-dimension dependency. The figures are very helpful in understanding how the proposed modules work. The experiment part covers quite some baseline models and datasets, demonstrating the method's superior performance. The rich ablation study covers multiple aspects of experiment settings, including padding, covariants, hyper-parameter selection and architecture designs. These experiment results proved that the modules introduced are critical to the final performance. Moreover, the author(s) analyzed the time complexity of the encoder & decoder, and further benchmarked the memory occupation and computation time of each baseline model against CrossFormer. The very detailed comparison convinced me of the model's effectiveness.\n\n  \n\nOn the other hand, the novelty of the paper is kinda limited because the problem of MTS forecasting and use of cross-dimension dependency was explored before, and Transformer was kind of the de facto in this setting. I also have a few concerns or questions as follows:\n\n1. In Figure 1, it's interesting to see the pattern demonstrated in the figure(a). Can you please further elaborate how that figure indicates the MTS data tends to be segmented?\n\n2. All the segments seem to be formed based on their location and segment length. Can you please explain how the segments align with the meaningful \"segments\" for forecasting in MTS data?\n\n3. In section 3.2, the author(s) proposed the design of cross-dimension stage after cross-time stage. Is there any performance comparison between the current design with the other way around?\n\n4. In sections 3.2, is there any performance comparison on the impact of using `LayerNorm` and skip connection?\n\n5. It's quite surprising that covariates don't improve the performance in ablation study, can you share the covariates used in your experiments?\n\n6. In the Hierarchical encoder decoder, the final results are summarized over each layer's output, is there any difference in the impact of each layer on the final results?\n\n  \n\nMinor suggestions:\n\n7. In section 3, the author(s) can mention the covariants are not used earlier in the paper to avoid confusion.\n\n8. In section 3.2, the author(s) used the notation of `L` to represent the number of segments, where $\\frac{T}{L_{seg}}$ was also used in complexity analysis. They could be unified if there's no difference in their meanings.\n\n  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "4NYrA6IoCxY", "reviewer": "Reviewer_Zk9T", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes Crossformer, a Transformer-based model utilizing cross-dimension dependency for Multivariate time series forecasting. Besides the time information, Crossformer considers dependency between dimension information. Specifically, Dimension-Segment-Wise (DSW) embedding and Two-Stage Attention (TSA) layer are proposed to efficiently capture both cross-time and cross-dimension dependency. Extensive experimental results show the effectiveness of Crossformer.", "summary_of_the_review": "This paper proposes a cross-time and cross-dimension model - Crossformer. However, efficiency is my main concern since Crossformer should perform 2-D attention and the paper doesn't provide any efficiency metric. Thus, I think the paper is not much practical.", "strength_and_weaknesses": "Strength: Dimension information is an interesting topic in the time series. This paper proposes Crossformer, which performs cross-time and cross-dimension dependency. The experiment results are comprehensive.\n\nWeaknesses: \n(i) Efficiency is my main concern about this paper. Since 2-D attention is performed in Crossformer, the complexity is much more than in the previous 1-D attention method. Can authors provide some metric of efficiency (i.e., FLOPs, number of parameters, time) in Table 1?\n(ii) Interestingly, there is another paper (https://openreview.net/forum?id=GpW327gxLTF) that investigates the dependency between dimensions. But their conclusion may be different from yours. For example, the dependency between ILI is significant compared with other data sets. However, in Table 1, Crossformer is not the best compared with other baselines on ILI. Can you explain that?\n(iii) The extension from 1-D to 2-D is somehow trivial in time series fields. Thus, the novelty of the paper is weak in my view.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-5%