CrossGNN: Confronting Noisy Multivariate Time Series Via Cross Interaction Refinement
Authors: Qihe Huang
Lei Shen
Ruixin Zhang
Shouhong Ding
Binwu Wang
Zhengyang Zhou
Yang Wang
Venue: NeurIPS 2023 poster (2023)
Keywords: Time Series Forecasting;

Abstract: Recently, multivariate time series (MTS) forecasting techniques have seen rapid development and widespread applications across various fields. Transformer-based and GNN-based methods have shown promising potential due to their strong ability to model interaction of time and variables. However, by conducting a comprehensive analysis of the real-world data, we observe that the temporal fluctuations and heterogeneity between variables are not well handled by existing methods. To address the above issues, we propose CrossGNN, a linear complexity GNN model to refine the cross-scale and cross-variable interaction for MTS. To deal with the unexpected noise in time dimension, an adaptive multi-scale identifier (AMSI) is leveraged to construct multi-scale time series with reduced noise. A Cross-Scale GNN is proposed to extract the scales with clearer trend and weaker noise. Cross-Variable GNN is proposed to utilize the homogeneity and heterogeneity between different variables. By simultaneously focusing on edges with higher saliency scores and constraining those edges with lower scores, the time and space complexity (i.e., $O(L)$) of CrossGNN can be linear with the input sequence length $L$. Extensive experimental results on 8 real-world MTS datasets demonstrate the effectiveness of CrossGNN compared with state-of-the-art methods.

CrossGNN: Confronting Noisy Multivariate Time Series Via Cross Interaction Refinement
Qihe Huang Lei Shen', Ruixin Zhang Shouhong Ding Binwu Wang', Zhengyang Zhouf Yang Wang" "
University of Science and Technology of China (USTC), Hefei, China Suzhou Institute for Advanced Research, USTC, Suzhou, China
Youtu Laboratory. Tencent, Shanghai, China Email: (hqh, whw 1995 @ wwa.l.sttc.du.com (7zy0929, angyan @ustc.edu.cn, shenlei19966 gmail.comm (ruixinzhang. ericshding] @tencent.com
Abstract
Recently, multivariate time series (MTS) forecasting techniques have seen rapid development and widespread applications across various fields. Transformer-basee and GNN-based methods have shown promising potential due to their strong ability to model interaction of time and variables. However, by conducting compre- hensive analysis of the real-world data, we observe that the temporal fluctuations and heterogeneity between variables are not well handled by existing methods. To address the above issues, we propose CrossGNN, a linear complexity GNN model to refine the cross-scale and cross-variable interaction for MTS. To deal with the unexpected noise in time dimension, an adaptive multi-scale identifier (AMSI) is leveraged to construct multi-scale time series with reduced noise. A Cross-Scale GNN is proposed to extract the scales with clearer trend and weaker noise. Cross-Varts Variable GNN is proposed to utilize the homogeneity and heterogene- ity between different variables. By simultaneously focusing on edges with higher saliency scores and constraining those edges with lower scores, the time and space complexity (i.e., (L)) of CrossGNN can be linear with the input sequence length T.. Extensive experimental results real-world MTS datasets demonstrate the effectiveness of CrossGNN compared with state-of-the-at methods. The code is available http://githbb.cooo//gg22////s//s
Introduction
Time series forecasting has been widely used in many fields (i.e., climate [ traffic 31 energy [3]. finance [13], etc) X 29 20, 10). Multivariate time series (MTS) consists of time series with multiple variables and MTS forecasting aims at predicting future values based on historical time series. Deep 17
learning models 2d
24
21 have demonstrated superior performance in MTS forecasting.
In particular, Transformer based models 34 27 35) have achieved great power in MTS, benefiting from its attention mechanism which can model the long-term interaction between different time points sequences (cross-time). Graph Neural Networks (GNNs) (28 [6 2 19 7 have also shown promising results for MTS forecasting, which can extract pre-defined or adaptive interaction between different variables (cross-variable)
However, a recent study 33 shows that simple linear model dramatically outperformed many state- of-the-art (SOTA) models, and inspires to investigate the reasons why the existing cross-time
This work is done when Qihe Huang intern YouTu Lab. X Yang Wang and Zhengyang Zhou corresponding authors.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

and cross- variable interaction modeling fail to enhance prediction performance. By conducting thorough analysis on real-world data, we observe that the presence of some unexpected noise (caused by humans, sensor distortion) may be responsible for it. In time dimension, as shown in Figure | (a). Transformer-basee basse model heavily relies on the input sequences to generate the attention map, while its prediction may be susceptible to incidental noise, even some small fluctuation (i.e., noise) could easily lead to significant shifts on temporal dependencies. Our findings reveal that self-attention mechanism tend to assign high scores to outlier points in time series, resulting in spurious cross-time correlations. In variable dimension, cross-variable correlation exhibits complex and dynamic evolution over time [18]. Despite the existence of underlying causal associations between variables, extracting cross- variable interaction is difficult due to the influence of noise interference. Additionally, as shown in F (b). we observe that such unexpected noise, which can be detected by the outlier detection algorithm [4]. accounts for a high proportion in the time series.
188o
Vaciaa
277M
(a)
(c)
(d)
Figure 1: Data analysis on the real-worke dataset [1]. (a) Forecasting results of Transformer- models suffering from unexpected noise. (b) Proportion of noise in the ETTh1, ETTm1, Traffic, and
Weather datasets (27 detected by [4]. (c) Different levels of noise signals multi-scale time series. (d) Homogeneous and heterogeneous relationships between variables.
Despite the non-negligible noise in time series, we can still discover the potential opportunities to confront noise challenges. (1) Cross-Scale Interaction. As shown on Figure (c), by performing multi-scale extraction on the time series, we observe that different scales possess distinct levels of noise intensity, typically with coarser scales exhibiting lower noise intensity. Evidently, captur- ing dependencies across different scales enables the cross-time relationships to be robust against noise [5]. (2) Cross-Variable Variable Interaction. As shown on Figure I (d), there is both homogeneity and heterogeneity for cross- varrible interaction in real-world data 36). In fact, the two relationships can both contribute to invariant connections during the temporal progression. Consequently, learning the invariant associations that contain both homogeneous and heterogeneous relationships among variables can boost its robustness to confront noise. Based on above analysis, it is still challenging to refine interaction in noisy MTS. The key obstacles be summarized as follows: 1) How to capture cross-scale interaction that is not sensitive to unexpected input noise. 2) How to extract cross-variable relations between heterogeneous variables.
In this work, we propose CrossGNN, which is the first GNN solution to refine both cross-time and cross-variable interactions for MTS forecasting. To deal with the unexpected noise in time dimension, we firstly devise an adaptive multi-scale identifier (AMSI) to construct multi-scale time series with different noise levels. In time dimension, propose Cross-Scale GNN, which is temporal correlation graph, to model the dependency between different scales. The scales with clearer trends and weaker noise will be assigned with more edge weight. In variable dimension, we first introduce heterogeneous interaction modeling between variables into MTS forecasting and propose cross- cross-variable GNN utilize the homogeneity and heterogeneity between different variables with positive and negative edge weights. By focusing on edges with higher saliency scores and constraining those edges with lower scores at the same time, CrossGNN achieves linear time and space complexity (i.c., O(L)) with input sequence length L. The main contributions are summarized as follows:
We conduct comprehensive studies on real-world MTS data and discover that the unexpected noise in time dimension and variable-wise heterogeneity between variables is not well handled by existing Transformer-based and GNN-based models.
We propose a linear complexity CrossGNN model, which the first GNN model to refine both cross-scale and cross-variable interaction for MTS forecasting.

---

1) To deal with the unexpected noise in time dimension, AMSI is leveraged to construct multi-scale time series with different noise level and Cross-Scale GNN is proposed to capture the scales with clearer trend and weaker noise.
2) Cross- Variable GNN is designed to model the dynamic correlations between different variables. This the first model to introduce heterogeneous interaction modeling between variables into MTS forecasting.
Cross-Variable
Extensive evaluation on 8 real-world MTS datasets demonstrates the effectiveness of Cross- GNN. Specifically, CrossGNN achieved top-1 performance on 47 settings and top-2 on settings when compared with 9 state-of-the-art models with varying prediction lengths.
2 Related Work
Multivariate Time Series Forecasting. MTS forecasting has witnessed significant advancements due to the emergence of deep neural networks. These networks can be based on Convolutional Neural
Network (CNN) 226 23 Recurrent Neural Network (RNN) 6) Transformer [1] 25| 27 35 9
Graph Neural Network (GNN) 28 32) Generally, the primary emphasis of these studies lies
in devising interactions between the temporal dimensions (cross-time) and the variable dimensions (cross-variable).
Cross-Time Interaction Modeling. Cross-time interaction modeling aims to capture correlations between different time points. Recently, CNN-based model TimesNet (26) transforms the time series into a two-dimensional matrix and uses a CNN-based backbone for feature extraction. RNN- based model LSTnet (6] utilizes the Long Short-Term Memory (LSTM) to model the temporal dependencies, but it may be limited by the inherent issue of gradient vanishing/exploding in RNNs. Transformer-baat based models benefit from its self-attention mechanism, enabling them to capture long- term cross-time dependency. AutoFormer (27) incorporates a decomposition mechanism that splits the input sequence into trend and seasonality. and integrate an coro-correlttion module into the transformer to capture long-term cross-time dependency. FedFormer [35] leverages a frequency- enhanced decomposition mechanism while incorporating additional frequency information. However, despite the outstanding performance of Transformer-basee methods, we observe that their self- attention mechanism susceptible to unexpected noise, as shown on Figure [a]. Based on these findings, we propose an innovative GNN-based method that constructs cross-scale temporal graph to mitigate the impact of temporal noise on modeling cross- pooss-time correlations.
Varsa-Va Interaction is proved to be critical for MTS forecasting (34, and numerous works have employed Graph Neural Networks (GNNs) 22 37 30 38 to capture cross-variable relationships. STGCN [32) firstly utilizes GNN to model the cross-variable dependency in traffic forecasting, which can effectively capture the dependency between different roads in pre-defined topology graphs. MTGNN (28) expands the utilization of GNN from spatio- temporal prediction to MTS forecasting, and proposes a straightforward method for computing adaptive vors-varrablle graph. On the other hand, the Transformer- -hased MTS prediction works have also recognized the potential of cross- variaale interactions to enhance prediction performance, such CrossFormer (34]. However, the cross-ver vooiibla relationship is dynamic and be greatly influenced by noise during the learning process. Given this, we refine the cross-vaie varrible relationship by decoupling homogeneity and heterogeneity in MTS, resulting in noise-insensitive relationship during the temporal evolution.
Cross- Variable Interaction Modeling.
3 Methodology
In long-term multivariate time series (MTS) forecasting, the input comprises historical sequences across D variables denoted by X' = {X ...., X 1/1 E R where L denotes the look-back window size and X denotes time series of the ith variate at the l4h time step. The objective of MTS
forecasting is to predict future time series denoted by X' {X
1 R TTTD based on
X, where T represents the prediction time steps and T > The detailed structure of CrossGNN is illustrated in Figure 2 We firstly employ an adaptive multi-scale identifier (AMSI) to generate multi-scale time series and reduce noise on coarse scale. Then, we construct scale-sensitive and trend- aware temporal graph to extract cross-scale interaction. We perform vrosslvariable aggregation via
3

---

Ouipet Madli-scale MIS X' o
Inpal of
Cross-scale interaction
Ouipun a gr
Clommel dimeemiio lisccar project
Mielly "M
Concarenate all scales in tinee dimenssio
prop
Scale Capture by A
Seale-snnshhve restriction
Trand awore selection
Madled cos oolat
Learnable vectur.
Iniolizen graph
pertod
Cross-variable interaction.
Hine Jostsin
scales
Time diinenion lincer project
Period Capture by FFT
PREMPIPREN
C. G
Cross-nnrable
amplitud variables
Impai
redats enligy dboesseedeoeent
Vartalle mede
Hben speer comtrtatioon
Ensalted grapa
Betrogreee corriation
(4) Adaptive multi scale identifier
(b) Cross interaction layer
(c) Forecasting
Figure 2: CrossGNN architecture. (a) Adaptive multi-scale identifier (AMSI) is devised to extract the multi-scale MTS X' from the input X. (b) Cross-Scale GNN facilitates cross- time interaction within and across different scales, while Cross-Variable Variable GNN models cross-variable interaction for both homogeneous and heterogeneous relationships. (c) Direct multi-step (DMS) forecasting is leveraged predict the future time series based on two MLPs.
modeling homogeneous and heterogeneous relationships between variables. Finally, direct multi-step (DMS) forecasting is adopted in decoder to predict future time series.
3.1 Adaptive Multi-Scale Identifier
The adaptive multi-scale identifier (AMSI) is designed to capture the different scales of MTS from coarse to fine, and reduce the unexpected noise on the coarse scale. Technically, we utilize Fast Fourier Transform (FFT) to analyze the time series in the frequency domain and calculate the potential periods of the time series, inspired by (26). We compute the amplitude of each time series at various frequencies using FFT and subsequently average the amplitude of X' across variable dimension:
A Avg. (Amp (FFT (X))),
where Amp(-) is the calculation of amplitude, FFT(-) is the calculation of FFT, R& represents the calculated amplitude of each frequency, which is averaged from D variables by Avg(-). We choose the frequencies {fi, f2, ... }ss} "ff. which correspond to the Top-S amplitude values:
(h..... Ss} arg Top-S(A):
where arg Top-S(-) picks out the S frequency values with the highest amplitude from A. The period are calculated by the selected frequencies as follows: }s} .ps lengths {PI.P2. Pa..P......
P.
Ja
8 8 (.)
(3)
Then, AvgPool(-) with kernel size pa and stride Ps. s 21, S). are applied to the MTS X' E RLXD in the time dimension to capture the MTS at 8-th scales:
X, = AvgPool(2 kernel=strideep..
where AvgPool(-) downsamples time series to obtain X, E RL())xD = is the time series length in the s-th scale, L.] is the operation of rounding down. We concatenate the captured different

---

scales in the time dimension and obtain a period-wise multi-scale MTS X' E Rtxx as the output of AMSI, L' - (s)) is the sum of lengths actoss all scales:
(5) Here, we employ an expansion dimension strategy (using an MLP), to create an embedding for each time step at different scale. This strategy is inherited from MTGNN (28), aiming to enhance the local semantics at each time step and positively impact subsequent cross-scale and cross-variable interactions. The shape of X' is finally expanded as R f.x DxC
X" Concat (K1, X22 Xz).
3.2 Cross-Scale GNN
The Cross-Scale GNN is designed to utilize the interaction of the multi-scale MTS X' and extract the scales with clearer association and weaker noise. The cross-scale graph in the time dimension is represented as Gseale (Vocale, Essalee) Vscale = {ufcale, ogcale ..... V/ usgale] is the time nodes set in all time scales, where v:tale is the i-th time node. gscate 1 assigns the correlation weights time
between each time node, each element in Esoule meaning the correlation weight between nodes (inter-scale or intra-scale). The key purpose of Cross-scale GNN is to learn the cross-scale temporal correlation weights Escale that insensitive to noise interference. To diminish the effect of noise on correlation weights, we maintain the independence by initializing Escule with the production of two learnable vectors vecicale and veciclae, ensuring Escale not affected by noise hidden in input:
Escale
x vec
where ReLU(() is the active function regularizing the weight matrix so that each element is positive, and (oftmax(-) is the operation to ensure the weights of all nodes correlated particular time node sum to 1.
Scale-sensitive Restriction. We consider that for any time node, the number of its correlated time nodes on the fine scale should be more than the number on the coarse scale. For any time node vecale, the number of its correlated nodes at s-th scale restricted to k2 = 1 where ps is the period length of s-th scale, [ denotes the ceiling function, K is a constant. This ensures that finer-scale time series contribute more temporal node associations. The neighbor time node at s-th scale (i.e. the correlated time nodes at s-th scale of uzcatc is denoted as:
p.
(7) where arg (-)- is an operation to extract k, nodes with highest correlation weight, Escate (uioaae) E [R)) is the correlation weight of time node vseate at s-th scale. In this way, the number of neighboring nodes at different scales can be restricted based on the correlation weight matrix Escale
Nacale (u:
(u. (oaae) = arg Top-ke (Egrole
Trend-aware Selection. To ensure the temporal trends can be captured, we preserve the associations between a time node and its both preceding and succeeding nodes. Denote Artrend ( (unrale) as the trend neighbor set of time node u. vooale, which can be defined as follows:
(8) where scale(-) provides the scale of time node. The trend neighbor set of ogoate consists of its adjacent time nodes (i.e., l - jl < 1) which share the same scale. This gives the ability to preserve temporal trends in the cross-scale correlation graph.
Atrend (veeale) = {u j I 1, scale(v.cale = scale peale(ogaulg) (v.cale))
Correlation Weight Re-normalization. Denote N(x::) = Nocale (v:) U. (v:) as the selected neighbor set of time nodes oxcale The final correlation weight re-normalized follows:
gacule
vf eN(v,,, otherwise,
Sccale [i
Σ ( Escule (1.m
(9)
0.
where [tcole(t, is the correlation weight between ooaal and uocate, This step filters out non- significant correlation and preserves restricted set of neighboring nodes A((ucalee to each node. Also, re-normalization applied to the retained correlation. Then, cross-scale correlation graph between time nodes at different scales is constructed.
5

---

Cross-scale interaction. After obtaining the cross-scale temporal correlation graph, we do cross- scale interaction in the time dimension based on GNN, and the information propagation process will be stacked for N layers:
HNV N. E,
E
E[i,5]
Hig Hosme,N +
E
Eli, m] Hoine
Hiima,N Hi:
a (Connaa (HLN
Hi: (
W)
Nime,
H4; /| 1|2
Nime,N time,N
Hi,
(10)
where the activation function; W is the learnable matrix; Htime is the time node feature; t the aggregation of neighbor time node feature. H.Aiiiee Ninme,N aggregates the neighboring node feature from the previous layer correlated to the time nodes of ufoule, Then H is updated by the
aggregated time node feature H.V i,; H.ttime,N and its previous layer feature Hi Hiime,N-1 Finally, the
normalized Hiime,N at N-th layer is the output of Cross-Scale GNN.
3.3 Cross- Variable GNN
Cross- Variable GNN designed extract the invariant correlation consisting of homogeneous and heterogeneous relationship. In variable dimension, the cross-vares voos-variable graph is represented as Geor (Vvar, Evar). Vear {vi nzaar, ..... D par) the variable node set, where is the number of variables, and u is the i-th variable node. Evar is the variable correlation weight matrix, each element Evar meaning the correlation weight between two variables. Evar initialized by production of two latent vectors vect and vec
E = (oftmax(ReLLU(aaq" x v)c)""))
(11)
Heterogeneity Disentanglement. Specifically, we select the nodes with K highest correlation weight as positive neighbors with homogeneous connections, and take the nodes with K lowest correlation score negative neighbors with heterogeneous connections. Denote E (u!ar) as the correlation weight variable nodes related to vieer, For variable ú; its two decoupled neighbor sets can be represented as Nueer (w:) Bottom-Kyth (gar( (u:)) and Araar (v;) Top-K yar (Evar (v:)). respectively.
Correlation Weight Re-normalization. correlation weights are derived as follows:
The corresponding homogeneous and heterogeneous
if UJ E Neer(v;),
eegerarr (wg) मoकlce E guani 3.d
Evar [i
Ev, extar(ug) Roor (pk)) vf E (oor(w:))
(12)
0.
otherwise
This process filters out edges other than homogeneous and heterogeneous edges. The weights of homogeneous edges are positively correlated with their correlation scores, while the weights of heterogeneous edges are negatively correlated with their correlation scores. Additionally, separate re- normalization is applied to the weights of homogeneous edges and heterogeneous edges, respectively. Then, the cross-varians graph is constructed with disentangled homogeneous and heterogencous correlations.
Cross-variable Interaction. be formulated as:
For variable Us the disentangled cross-variable message passing can
H.N N
Evar Evar(i,] P +
Evar ii, H. N-1
"EEAaaT(a) (o4)
EVver (v:)
(13)
Mar,N 1,
(Concat(HLN"") N H. N1 W).
H
H /|| 112,

---

Huar is the variable node feature; HN is the aggregation of neighbor variable node feature. HM aggregates both homogeneous and heterogeneous neighbor node feature of previous layer correlated to waar H: is updated by the aggregated variable feature H.N and its previous layer feature H5 HaarNN1 Finally, the normalized H. at N-th layer is the output of Cross-variable GNN.
3.4 Direct Multi-step forecasting
After obtaining the output features of Cross- Variable GNN, we exploit the direct mteti-step (DMS) forecasting [33] for the decoder to predict the multi-step MTS at once. We take two MLPs as the decoder, where the first MLPc c maps the time dimension of features from C to 1, while the second MLPT maps the time dimension from the historical input sequence T,' to the output sequence length.
The final prediction can be obtained by:
(Xi, T. X
(MLPP(HHaar)
(14)
Table I: MTS forecasting results in terms of MSE and MAE, the lower the better. The prediction
length T' E 196, 192, 336, ]20} and look back window size is set as 96. The best results are highlighted in bold and the second best are underlined.
Mosiih Mettic
CraSGNN
TanceNcc
Canalomece
PaidITT
ETStoreee
Diiincse MSSE
FED
Autadoema
Pyniooeee
MTTGNN
MSB
MAE
MSE
MAE
MSE
MAB
MSE
MAB
MSB
MAE
MAB
MSE
MAL
MSB
MAE
MSL
MAE:
MSE
MAB
Bleomnary E771 ETTT E7Tm22 ElTall
0.373
0.376
0.375
0.408
0.598
0.345
0.380
0.372
0.389
0.379
0.419
0.505
1.553
0.475
0.543
0.51
336
0.417
0.445
0.459
0.557
0.754
0.499
0.453
0.543
0.490
0.561
96
0.176
0.266
0.167
0.367
0.308
0.202
0.192
0.273
0.189
0.250
0.108
0.202
0.303
0.287
0.255
0.350
0.455
0.507
0.208
0.209
192
0.240
0.397
1229
0.309
0.563
0.312
01252
0,314
0.255
0.319
0.54
0,302
0.269
0.328
1.251
0.340
0.730
0.675
0.255
0.324
3.5
0.501
0.345
0.321
0.55
0.337
0.309
0.318
0.357
0.314
0.357
0.39
0.427
0,325
0.365
1.339
03312
1.201
0.845
0.365
0.314
720
0.011
0.400
0.408
07401
0.429
0.410
0.413
0.416
0.414
0.415
0.54
0.522
0,421
0.415
0.433
0.43
3,025
1.451
0.461
0.455
96
0.582
0.299
0.364
0.411
0.386
0.828
0.385
0.408
0.041
0.679
0.366
0.400
0.376
0.419
11:49
0.5%
0.64
0.61
0.15
0:17
192
0.411
0.538
0.50%
0457
03352
0:426
0.448
1,500
0
0.700
0.681
336
0.465
0.485
0.62
0.574
0.521
0461
0.459
0.489
0.465
1.311
0,466
0.991
0.62
0.562
0519
0.516
0.506
0.507
0514
0.517
96
0,319
0.359
0.340
0.391
0.343
0.3%
0,50
033
0.387
0,358
0.997
0,366
0.58
0.055
0.597
0.354
0.454
192
0,900
(1.80
0.00
04414
0,419
0.86
0.630
0.639
0.466
0.429
0.439
1.266
0481
07155
0.88
335
0.436
0.444
0.0
0,449
0.465
0.418
0.40
0.465
0.47
0.594
0.541
0.4%
0.487
1283
1466
0507
0.747
0.515
0.540
720
0.445
0.464
0.02
0.468
0.479
0.505
0.464
0.483
0.300
0.497
0.931
0.657
0.463
0.474
13515
0511
0.933
0.785
0.532
0.576
96
0.133
0275
0158
0.722
0.185
0.258
0.268
0:187
0.50%
0.197
0.282
0.193
0.358
11201
02217
0385
0.49
0.17
0.318
192
0195
0258
0154
0289
0,501
0.205
0:23
0.109
0365
0.966
0.285
0.201
0.355
11222
0375
0.45
0238
0.352
180
02700
( 200
0:103
0:11
0.112
0:706
0312
0.339
0.00
0.00
0:314
1121
018
0326
6280
720
0221
0.531
0 0.315
11218
03317
0.231
0.545
0.345
0.331
0:346
0.355
11254
0.36
0.376
0.445
1250
TTilicc
191
8317
0317
0.007
0.399
0.5%
0.370
0.604
0.375
11013
0.582
0.369
356
0.588
0334
0.330
0.03
0.396
0.605
0.373
0.355
1163
0.557
0.581
0.669
0.597
0.33
0840
0,350
0.600
0.346
B60
833
0.833
0.566
0.45
0.394
0.666
0.82
0.600
0.08
0.395
0473
0.45
bachonge Washhrr
96
8.159
0.218
0.12
0.220
0.191
0.251
L1T
0.230
0.197
0.281
0.196
0.255
0.117
0.296
0.266
0.336
0.22
0.556
0.230
0.329
193
0.211
(1266
0010
0.361
0.119
0.219
1220
0.221
0.237
0.12
0257
0.376
0316
1307
11360
0.739
0.644
0.363
0.32
336
0.267
11310
(1200]
0.306
03357
0.832
1277
0.321
0.206
0.53
0233
0:335
0.339
0380
1339
1.395
1.004
0.755
0.344
0.396
720
0.252
01302
0.303
0.259
0,368
0.5%%
11365
0.367
0.352
0.388
0.345
0.311
0.403
0.628
1.419
0.428
1.400
0.934
0.009
0.371
0.107
0.214
0.108
0.223
0.005
0.204
0.088
0218
0.145
0.278
1197
0.323
1,745
1,105
0.102
192
6.171
0.310
0.197
0.316
0.181
0.505
0.176
0.315
0.271
0.380
1,300
0.309
1.574
1.151
720
8.815
0.773
1.025
0.39
0.685
1.395
0.841
1.447
0.941
2.085
4 Experiment
4.1 Datasets And Experiment Setup
Datasets We conduct extensive experiments on 8 real-world datasets following (27. including Weather, Traffic. Exchange Rate, Electricty and 4 ETT datasets(ETThl, ETTh2, ETTml and ETTm2). We follow the standard protocol in (27 and split datasets into training, validation and test set by the ratio of 6:2:2 for the last 4 ETT datasets, and 7:1 ::22 for the other datasets.
Baselines and Setup We compare our method with 9 state-of-the-ar methods, including Times- Net (26): 6 Transformer-basee
methods: PStchTST [12), Crossformer [34]. ETSformer [23]. FEDformer (55) Pyraformer [[], Autoformer 277; GNN-based method: MTGNN 28;; simple
yet powerful linear model Dlinear 33. All the models are following the same experimental setup
with prediction length 7 E (96, 192, 336, 720} for all datasets as in the original papers. We collect
all bascline results from [26] except MTGNN, with the default look-back window L 96. We
7

---

Table 2: Performance comparisons on ablative variants
Datasets
Weather
Traffic
ETTm2
Predict Length
96
192
336
720
96
192
336
720
96
192
336
720
C-AMSI
MSE
0.167
0.220
0.276
0.358
0.585
0.590
0.596
0.610
0.192
0.249
0.313
0.420
MAE
0.229
0.284
0.318
0.382
0.330
0.336
0.341
0.354
0.279
0.318
0.358
0.417
MSE
0.175
0.231
0.290
0.371
0,588
0.599
0.603
0.614
0,200
0.259
0.328
0.422
C-CS
MAE
0.241
0.286
0.325
0.385
0.331
0.343
0.342
0.358
0.281
0.325
0.364
0.421
C-Hete
MSE
0.172
0.224
0.277
0.364
0.589
0.588
0.603
0.616
0,190
0.255
0.320
0.417
MAB
0.231
0.280
0.327
0.381
0.329
0.332
0.335
0.354
0,280
0.320
0.358
0.417
C-CV
MSE
0.174
0.227
0.279
0.367
0.588
0.591
0.604
0.616
0.193
0.257
0.322
0.419
MAE
0.235
0.281
0.328
0.382
0.330
0.335
0.338
0.358
0.278
0.323
0.359
0.418
CrossGIN
MSE
0.159
0.211
0.267
0.352
0.570
0.577
0.588
0.597
0.176
0.240
0.304
0.406
MAE
0.218
0.266
0.307
0.362
0.310
0.321
0.324
0.337
0.265
0.307
0.345
0.400
reproduced the results of MTGNN [88] with look-back window L 96 on all datasets according to the settings in the original paper. We calculate the Mean Square Error(MSE) and Mcan Absolute Error(MAE)) of MTS forecasting as metrics. More details about datasets, baselines, implementation. syper-parammttrs are shown in Appendix A.3
4.2
Main Results
The quantitative results of MTS forecasting using different methods is shown in Table D CrossGNN achieves outstanding performance on most datasets across various prediction length settings, obtaining 47 first-place and 9 second-place rankings in total 64 settings. Quantitatively, compared with the best results that Transformer-base
methods can offer, CrossGNN achieves an overall 10.43% reduction on MSE and 10.11% reduction on MAE. Compared with GNN-based method MTGNN, CrossGNN achieves a more significant reduction 22.57% on MSE and 25.74% 74% on MAE. Compared with other strong baselines like TimesNet and Dlinear, CrossGNN can still outperform them in general. Our method dose not achieve the best performance on the Electricity dataset. Further analysis reveals that the more severe -iitriiiiiitttionn (OOD) problem in Electricity dataset results in a lower generalization of the learned temporal graph relations on the test set.
4.3 Robustness Analysis of Noise
To evaluate the model robustness against noise, we add different intensities of Gaussian white noise to the original MTS and observe the performance changes of different methods. B shows the MSE results of CrossGNN, ETSformer
25 and MTGNN (28) under dif-
ferent noise ratio on ETTm2 dataset, and the input length is set as 96. As the signal-to-noise ratio (SNR) decreases from 100db to Odb, the mean square error (MSH) increases more slowly on CrossGNN (0.177) compared to ETSformer (0.191) and MTGNN (0.205). The quantita- tive results demonstrate that CrossGNN exhibits good robustness against noisy data and has great advantage when dealing with unexpected fluctuations. We speculate such improvements benefit from the explicit modeling of respective scale-level and variable-level interactions.
0.6
Cro
ETSfomeer
0.5
MIGNN
MSF
0.4
0.3
0.2
10
10a
Figure 3: Robustness analysis under different signal-to-noise ratio (SNR) on ETTm2.
4.4 Ablation Study
We conduct ablation studies by removing corre-
sponding modules from CrossGNN on three datasets. C-AMSI removes the adaptive multi-scale identifier (AMSI) and directly use k fixed lengths (e.g., 1, 2, ...,, k) for average pooling. C-CS removes the Cross-Scale GNN module. C-Hete removes the heterogeneous connections and focuses on the homogeneous correlation modeling between different variables. C-CV removes both homoge- neous and heterogencous connections. We analyze the results shown in Table 2 Obs.1) I) Removing
8

---

Cross-Scale GNN results in the most significant decrease in prediction metrics, emphasizing its strong ability in modeling the interaction between different scales and time points. Obs.2) Cross- Variaale GNN also improves the model performance a lot, demonstrating the importance of modeling the complex and dynamic interaction between different variables. Obs.3) AMSI constantly improves the forecasting accuracy, suggesting that different scales of MTS contain rich interaction information.
4.5 Hyper-Paramett sensitivity
Look-Back Window Size Figure 4 shows the MSH results of models with different look-back window sizes four datasets. As the window size increases, the performance of Transformer-basee models fluctuates while CrossGNN constantly improves. This indicates that the attention mechanism of Transformer-based methods may focus much more on the temporal noise but our method can better extract the relationships between different time nodes through Cross-Scale GNN.
Number of Scales We vary the number of scales from 4 to 8 and report the MSE and MAE results on Weather and Traffic dataset. As shown in Figure [ (a) and Figure 5 (b), We observe that the performance improvement becomes less significant after certain number of scales, indicating that certain scale size is sufficient to eliminate most of the effects of temporal noise. Number of Node Neighbors The number of neighboring nodes limited each time node is mainly determined by the hyperparameter K. As shown on Figure (c) and Figure [5 (d), we experiment with K values of 10, 15, 20, 25, and 30 and found that CNossGNN is not sensitive to the number of K. This indicates that it is only necessary to focus on strongly correlated nodes for effective information aggregation in temporal interaction.
0.58
04414
0:10-
0,33
0.00
CrossGNN
Dlincar
Autofommer
FEDooomer
ETSfomer
Figure 4: The MSE results (Y-axis) of models with different look-back window sizes (X-axis) ETTh2, ETTm2, Traffic and Weather, the output length set as 336.
Trallc
0.34
(9)
(c)
(4)
Figure The MSE (left Y-axis) and MAE results (right Y-axis) of CrossGNN on Traffic and Weather. (a) and (b) display the performance on different scale numbers. (c) and (d) demonstrates the performance on different number of node neighbors
4.6 Complexity Analysis
Table 3 illustrates the theoretical complexity of CrossGNN and existing Transformer-base methods. Detailed complexity derivation can be found in the Appendix C. To verify that the time and space complexity of our method is indeed (L), we use TVM to implement the GNN computation part and compare the computation time and memory usage of with full connected graph during inference on ETTh2. Comparison experiments are implemented on Intel(R) 8255C CPU @ 2.50GHZ with 40GB memory, centos 7.8, and TVM 1.0.0. Figure [6) illlsttrtss the time and memory cost of GNN modules, and our proposed approach is close to linear with the input length.

---

Method
Complexity per layer
1800
Cerpl.
190
160
Gal
Transformer
O(L²)
1400
120
Informer
Oo L)
1200
Autoformer OL L) (L))
510
Pyraformer
CrossGNN
O(L)
Figure 6: Comparison of time and memory con-
Table 3: The computation complexity in the- ory. represents the length of the input data.
sumption on ETTh2. (a) memory occupation. (b) computation time.
5 Conclusion and Future Work
In this work, we construct a comprehensive analysis of real-world data and observe that the temporal fluctuations and heterogeneity between variables, caused by unexpected noise, are not well handled by current popular time-series forecasting methods. To address above issues, we propose a linear complexity CrossGNN model, which the first GNN model to refine both cross-scale and cross- variable interaction for MTS forecasting. An adaptive multi-scale identifier (AMSI) is leveraged to obtain multi-scale time series with reduced noise from input MTS. In particular, Cross-Scale GNN captures the scales with clearer trend and weaker noise, while Cross Varoiala GNN maximally exploits the homogeneity and heterogeneity between different variables. Extensive experiments on real-world MTS datasets demonstrate the effectiveness of CrossGNN over existing SOTA methods while maintaining linear memory occupation and computation time as the input size increases. For future work, it is worth exploring the design of dynamic graph networks that can effectively capture complex interactions in duitt--distribution (OOD) scenarios.
6 Acknowledgement
This paper is partially supported by the National Natural Science Foundation of China 2No.62072427, No. 12272217,, the Project of Stable Support for Youth Team in Basic Research Field, CAS (No.YSBR-OO YSBR-005), Academic Leaders Cultivation Program, USTC.
References
[1] Rafal A Angryk, Petrus C Martens, Berkay Aydin, Dustin Kempton, Sushant S Mahajan, Sunitha Basodi, Azim Ahmadzadeh, Xumin Cai, Soukaina Filali Boubrahimi, Shah Muhammad Hamdi, al. Multivariate time series dataset for space weather data analytics. Scientific data, 7(1):227, 2020.
[2] Dawei Cheng, Fangzhou Yang, Sheng Xiang, and Jin Liu. Financial time series forecasting with multi-modality graph neural network. Pattern Recognition, 121:108218, 2022.
[3] Ömer Fahrettin Demirel, Selim Zaim, Ahmet Çalişkan, and Pinar Özuyar. Forecasting natural gas consumption in istanbul using neural networks and multivariate time series methods. Turkish Journal of Electrical Engineering and Computer Sciences, 20(5):695-711, 2012.
[4] Zijie Fang, Yang Chen, Yifeng Wang, Zhi Wang, Xiangyang Ji, and Yongbing Zhang, Weakly- supervised semantic segmentation for histopathology images based on dataset synthesis and feature consistency constraint. Proceedings of the AAAI Conference on Artificial Intelligence. 37(1):606 613,, Jun. 2023.
[5] Hansika Hewamalage, Christoph Bergmeir, and Kasun Bandara. Recurrent neural networks for time series forecasting: Current status and future directions. International Journal of Forecasting, 37(:))388 427, 2021.
[6] Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In The 41 international ACM SIGIR conference on research & development in information retrieval, pages 95-104, 2018.
10

---

[7] Jianxin Li, Qingyun Sun, Hao Peng, Beining Yang, Jia Wu, and S Yu Phillp. Adaptive subgraph neural network with reinforeed critical structure mining. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[8] Zhe Li, Shiyi Qi. Yiduo Li, and Zenglin Xu. Revisiting long-term time series forecasting: An investigation on linear mapping. ArXiv, abs/2305. 10721, 2023.
[9] Shengsheng Lin, Weiwei Lin, Wentai Wu, Songbo Wang, and Yongxiang Wang. Petformer: Long-term time series forecasting via placeholder-eehanceed transformer, 2023.
[10] Shengsheng Lin, Weiwei Lin, Wentai Wu, Feiyu Zhao, Ruichao Mo, and Haotong Zhang. Segrnn: Segment recurrent neural network for long-term time series forecasting, 2023.
[11] Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li, Weiyao Lin, Alex X Liu, and Schahram Dustdar. Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting, In International conference on learning representations.
[12] Yuqi Nie, Nam H. Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam, A time series is worth 64 words: Long-tem teng-termmnn forecasting with transformers. In International Conference on Learning Representations, 2023.
[13] Andrew Patton. Copula methods for forecasting multivariate time series. Handbook of economic forecasting, 2:899-960, 2013.
[14] P.J. Roussceuw and A.M. Leroy, Robust regression and outlier detection. Wiley StatsRef: Statistics Reference Online, 1:1-27, 2009.
[15] Amin Shabani, Amir Abdi, Lili Meng, and Tristan Sylvain. Scaleformer: Iterative multi-scale refining transformers for time series forecasting. The Eleventh International Conference on Learning Representations, 2023.
[16] Chao Shang, Jie Chen, and Jinbo Bi. Discrete graph structure learning for forecasting multiple 2021.
time series. arXiv preprint arXiv:2
[17] Bowen Shen, Zheng Lin, Yuanxin Liu, Zhengxiao Liu, Lei Wang, and Weiping Wang. COST- EFF: Collaborative optimization of spatial and temporal efficiency with slenderized multi-exit language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1719-1730, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.
[18] Qingyun Sun, Jianxin Li, Hao Peng, Jia Wu, Xingeheng Fu, Cheng Ji, and S Yu Philip. Graph structure learning with variational information bottleneck. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pages 4 4165-4174, 2022.
[19] Qingyun Sun, Jianxin Li, Haonan Yuan, Xingcheng Fu, Hao Peng, Cheng Ji, Qian Li, and Philip Yu. Position-aware structure learning for graph topology-imbalance by relieving under-reaching and over-squashing In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, pages 1848 1857, 2022.
[20] Tianyu Tang, Jianquan Lu, and Jianlong Qiu. Robust finite-time consensus coupled markovian logical dynamic networks. IEEE Transactions Circuits and Systems I1: Express Briefs, 2023.
[21] Hao Tian, Xiaolong Xu, Lianyong Qi, Xuyun Zhang, Wanchun Dou, Shui Yu, and Qiang Ni. Copace: Edge computation offloading and caching for self-driving with deep reinforcement learning. IEEE Transactions on Vehicular Technology, 70212):13281-11223 2021.
[22] Binwu Wang. Yudong Zhang, Xu Wang, Pengkun Wang, Zhengyang Zhou, Lei Bai, and Yang Wang. Pattern expansion and consolidation on evolving graphs for continual traffic prediction. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 2223-2222222222222 2232, 2023.
[23] Huiqiang Wang. Jian Peng, Feihu Huang, Jince Wang, Junhui Chen, and Yifei Xiao. Micn: Multi-scale local and global context modeling for long-term series forecasting. The Eleventh International Conference on Learning Representations, 2023.
11

---

[24] Xu Wang, Hongbo Zhang, Pengkun Wang, Yudong Zhang, Binwu Wang, Zhengyang Zhou, and Yang Wang. An observed value consistent diffusion model for imputing missing values in multivariate time series. In Proceedings of the 29th ACM SIGKDD Conference Knowledge Discovery and Data Mining, pages 1440---188,,, 2023.
[25] Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven C. H. Hoi. Etsformer: Exponential smoothing transformers for time-series forecasting. 2022.
[26] Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long, Timesnet: Temporal 2d-variation modeling for general time series analysis. In International Conference on Learning Representations, 2023.
[27] Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long, Autoformer: Decomposition trans- formers with correlooril for long-term series forecasting. Advances in Neural Information Processing Systems, 34:22419-22430, 2021.
[28] Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang, Xiaojun Chang, and Chengqi Zhang. Connecting the dots: Multivariate time series forecasting with graph neural networks. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2020.
[29] Xiaolong Xu, Xuyun Zhang, Honghao Gao, Yuan Xue, Lianyong Qi, and Wanchun Dou. Become: Blockchain-enablee computation offloading for iot in mobile edge computing. IEEE Transactions on Industrial Informatics, 16(6):4187-44 4195, 2020.
[30] Kuo Yang. Zhengyang Zhou, Wei Sun, Pengkun Wang, Xu Wang. and Yang Wang. Extract and refine: Finding support subgraph set for graph representation. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 2953-2964, 2023.
[31] Yi Yin and Pengjian Shang. Forecasting traffic time series with multivariate predicting method. Applied Mathematics and Computation, 291:266-278, 2016.
[32] Bing Yu, Haoteng Yin, and Zhanxing Zhu. Spatio-temporal graph convolutional networks: A deep learning framework for traffic forecasting. In Proceedings of the 27th International Joint Conference on Artificial Intelligence (IJCAI), 2018
[33] Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? arXiv preprint arXiv:2205. /3504 2022.
[34] Yunhao Zhang and Junchi Yan. Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting. In International Conference on Learning Representa- lions, 2023.
[35] Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. FEDformer: Frequency enhanced decomposed transformer for long-term series forecasting. In Proc. 39th International Conference on Machine Learning (ICML 2022), 2022.
[36] Zhengyang Zhou, Qihe Huang, Gengyu Lin, Kuo Yang, LEI BAI, and Yang Wang. GReto: Remedying dynamic graph topology-task discordance via target homophily. In The Eleventh International Conference on Learning Representations, 2023.
[37] Zhengyang Zhou, Qihe Huang, Kuo Yang, Kun Wang, Xu Wang, Yudong Zhang. Yuxuan Liang, and Yang Wang. Maintaining the status quo: Capturing invariant relations for ood spatiotemporal learning. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD '23, page 3603-3614, New York, NY, USA, 2023. Association for Computing Machinery.
[38] Zhengyang Zhou, Yang Wang, Xike Xie, Lianliang Chen, and Hengchang Liu. Riskoracle: minute-level citywide traffic accident forecasting framework. In Proceedings of the AAAI conference artificial intelligence, volume 34, pages 1258-1265, 2020.
12

---

Appendix
A Experimental Details
A.1
Datasets
We conduct extensive experiments on 8 real-world datasets following 127] The interval length, time step number, and the variable number of each real-world dataset are presented Table 4 The detailed dataset descriptions are as follows:
ETTh (ETTh1, ETTh2, ETTm1, ETTm2) consists of two hourly-level datasets (ETTh) and two
I Sminute-level datasets (ETTm). Each of them contains seven oil and load features of electricity transformers from July 2016 to July 2018.
2) Weather includes 21 indicators of weather, such as air temperature, and humidity. Its data is recorded every 10 min for 2020 in Germany.
3) Traffic describes hourly road occupancy rates measured by 862 sensors on San Francisco Bay area freeways from 2015 2016.
4) Exchange-rate collects the daily exchange rates of 8 countries from 1990 to 2016.
5) Electricity contains hourly electricity consumption (in Kwh) of 321 clients from 2012 to 2014.
Table 4: The statistics of the datasets for MTS forecasting.
Datascts
ETTh1
ETTh2
ETTml
ETTm2
Weather
Traffic
Exchange-rate
Electricity
Interval Length
Hour
Hour
15 Minutes
15 Minutes
10 Minutes
Hour
1 Day
Hour
Time step 17,420
17,420
69,680
69,680
52,696 21
17,544
7,588
26,304
Variable
7
7
862
8
321
A.2 Baseline Methods
We briefly describe the selected 9 state-of-the-art baselines as follows:
1) TimesNet [26] is a task-general foundational model for time series analysis that utilizes a mod- ular architecture to unravel intricate temporal variations. A parameter-efficient inception block is leveraged to capture intra-period and inter iiterperrood variations in 2D space.
2) Crossformer ables.
34
is versatile vision transformer which solves attention among different vari-
3) PatchTST tation.
12 is strong baseline dependent channel-independence and time series segmen-
25 is a Transformer-based model for time-series forecasting that incorporates inductive biases of time-series structures and introduces novel exponential smoothing attention (ESA) and frequency attention (FA) improve performance.
ETSformer
5) DLinear
33 is a simple linear-based model combined with a decomposition scheme.
6) FEDformer 35 is Transformer-ben bassd model that uses the seasonal -teend decomposition with erehueeey-enhanced blocks to capture cross-time time dependency for forecasting.
model using decomposition architecture with an auto- Correlation mechanism sm to capture cross-time dependency for forecasting.
7) Autoformer 22 is a Transformer-base
8) Pyraformer
a Transformer- based model learning multi-resolttion representation of the time series by the pyramidal attention module to capture cross-time dependency for forecasting. 9) MTGNN (28) explicitly utilizes cross-variable dependency using GNN. A graph learning layer learns graph structure where each node represents one variable in MTS.
A.3 Implementation Details
To ensure a fair comparison, the look-back window size is set to 96, which is consistent with all baselines. We set the scale numbers have shown that S does not have a significant impact beyond 5 and CrossGNN is not sensitive to K. Additionally, the mean squared error (MSE) used as loss function. For the learning rate, a grid search is conducted among [5e-3, le-3, 5e-4, le-4, 5e-5, le-5] to obtain the most suitable learning rate for all datasets. Besides, the dimension of the channel set as for smaller datasets and for larger datasets, respectively. The training would be terminated early if the validation loss does not
and set K to 10 for all datasets, as sensitivity experiments
13

---

decrease for three consecutive rounds. The model is implemented in PyTorch 1.8.0 and trained on single NVIDIA Tesla V100 PCIe GPU with 16GB memory.
B
Additional Experimental Results
B.1 Analysis on Robustness Against Noise
ETTIC
Weather
ETTm2
COLAONN ETSS0mee MIGNN
CrosGNN ETS
COIOONN
MICON
6.3
633
0.11
Figure 7: The MSE results (Y-axis) of models on ETTh2, ETTm2 and Weather with different signal- to-noise ratio (SNR).
To evaluate the robustness of CrossGNN against noise, add different intensities of Gaussian white noise to the original MTS and observe the performance changes. As the intensity of Gaussian white noise increases, the signal-to-noise ratio (SNR) gradually decreases from 100 dB to dB. Figure| shows the MSE results of CrossGNN, ETSformer [25] and MTGNN (28) on ETTh2, ETTTm2, and Weather under different SNR. As the SNR decreases from 100db to Odb, the mean square error (MSE) of CrossGNN increases more slowly than MTGNN and ETSformer.
Taking the results the ETTm2 dataset an example, when the noise intensity increases the beginning (i.e., SNR decreases from 100db to 10db), the prediction accuracy of MTGNN and ETSformer becomes unstable. Their prediction accuracy drops more rapidly when the noise intensity suddenly increases (i.e., SNR decreases from 10db to Odb). In contrast, CrossGNN maintains overall stability, and its performance degrades more slowly. The quantitative results demonstrate that CrossGNN exhibits good robustness against noisy data and has great advantage when dealing with unexpected fluctuations. Such improvements benefit from the explicit modeling of respective ccoss-scale and cross-vaud varsall interactions.
B.2 Sensitivity Analysis
MAAE (DAAA T-Asl)
21Ta2
Tmal::
011
6,00 012
(d)
Figure 8: The MSE (left Y-axis) and MAE results (right Y-axis) of CrossGNN with different number of scales (X-axis) on ETTh2, ETTm2, Traffic, and Weather.
Number of Scales We vary the number of scales from 4 and report the MSE and MAE results on ETTh2, ETTm2, Traffic, and Weather. As shown in Figure We observe that the performance improvement becomes less significant after a certain number of scales (i.e., 5), indicating that certain scale size is sufficient to eliminate most of the effects of temporal noise.
Number of Temporal Node Neighbors The number of temporal neighboring nodes primarily determined by the hyperparameter K. As depicted in Figure g we conducted experiments with
14

---

SAA (RUAA T-A300
Tudlec
5.0
0.74
0.70
6.30
(U)
Figure 9: The MSE (left Y-axis) and MAE results (right Y-axis) of CrossGNN with different K (X-axis) ETTh2, ETTm2, Traffic, and Weather.
different K values, including 10, 15, 20, 25, and 30, and observed that CrossGNN is not sensitive to the number of K. This suggests that effective cross-scale interaction can be achieved by focusing only strongly correlated time nodes.
B.3 Visualization of Forecasting Results Different Models
We present the visualization of forecasting results of CrossGNN and other baseline models on 8 datasets Figure and Figure [1] These datasets exhibit diverse temporal patterns, with 96-steps input length and output horizon. It can be observed that the prediction results of the Transformer- based model are significantly affected by noise, resulting in fluctuations. contrast, the prediction results of CrossGNN are less affected by noise, and the predicted values are closer to the true results.
For example, considering the forecasting results on the Traffic dataset, there are three unexpected noise points (i.c., irregularly high points) in the input data. During prediction, the attention mechanism of the Transformer-ban based model may focus on the noisy points, leading to a bias towards higher output predictions. As result, although the Transformer-basee model seems to capture the periods of the time series, fails to produce accurate predictions. In contrast, CrossGNN unaffected by these three noisy data points and generates predictions that are closer the ground truth. While Transformer-bar models struggle to capture the scale and bias of future data due to unexpected noise in the input data, CrossGNN outperforms other models in terms of both scale and bias in forecasting.
C Derivation of Computational Complexity
In this section, we theoretically prove that the time and space complexity of the Cross-Scale module and Cross-Variabee Variable module in CrossGNN are both linear. We have organized the notations used in Table for ease of reading.
Table 5: Mcaning of notations
Notation
Meaning
The i-th time node
Number of scales
Index of the scale Period length of the s-th scale Original input length (i.e., look-back window size)
L
L(s)
Time length at the s-th scale
L'
Total length of concatenated multi-scale time series
Ksoale
The hyperparameter to control temporal neighbor numbers
Kvar
The hyperparameter to control variable neighbor numbers
The temporal neighbor numbers at the s-th scale
A(e))
Total temporal neighbor node number correlated to U;
Total correlated temporal node pair number
D
Variable numbers
15

---

GroundTruth
Informer
Fedformer
Autoformer
CrossGNN
1.5
1.0
0.6
0.0
-0.5
0.0
-02-
-1.5
-0.4
50
100
125
150
175
200
50
125
175
200
llctrrcity
Exchange-ratee
0.75
0.00
-025
0050
-0.75
-1.00
-1
50
1000
150
175
25
125
150
200
TrafTic
Weather
Figure 10: Visualization of 96-step forecasting results on Electricity, Exchange-rate, Traffic, and Weather, and the look-back window size is set as 96.
Proposition 1. The time and space complexity for the Cross-scale GNN is O(Kscale x In L) and amounts to (L)) when S and K Kacale are constants w.r.l L.
Proof. To improve readability, substitute K for Kseate, Denote L(s) the number of time nodes at s-th scale:
- - ps
L(s)
(15)
where is the corresponding period length of the s-th scale and L is the original input length (i.e., look-back window size). L' the sum of time nodes different scales, and it could be expressed by:
S I'
S
Σ L(s)
Σ
L(nn+cc 25
(16)
Ps
where InS + +
25 ~ E.
1/2 is the approximate summation formula for the harmonic series, and
is the
constant.
For time node, we scale-sensitive time node neighbor numbers k p. s-th scale. Since the trend-aware neighbor nodes are defined as its previous node and next node the current scale, the number of trend-aware neighbor nodes can reach 2 when these nodes do not overlap with scale-sensitive neighbor nodes. However, when there is overlap, the number of trend-aware neighbor nodes can be 0 or I. I.Tereffore, the maximum neighbor node number of v, is given by:
16

---

GroundTruth
Informer
Fedformer
Autoformer
CrossGNN
13
1.0
A1
0.5
0.0
0.
-1.5
-2.0
50
75
00
125
150
175
200
50
75
100
125
150
175
200
ETTh1
E1Th2
1.0
0.5
0.3
-1.0-
-2
-1.5-
-2.9
50
125
175
25
200
ETTm
ETTm2
Figure 11: Visualization of 96-step forecasting results on ETTh1, ETTh2, ETTml and ETTm2, and the look-back window size is set as
S
A(v,)
S
kg
+
(17)
8=1
$=8
K
+2
(18)
8=1 p(s)
1
K + +...
5)
+
(19)
5
~ K(n S +e+
+
(20)
28
Total correlated node pair number is expressed as:
A= I,' x A(w;) S L(In e + 2/5 (K
1
+ 2) v 2K x lnn(S) L.
(21)
25
Consequently, the complexity of the proposed cross-scale GNN is:
((A) < 0(2K In (S) x L).
(22)
Since K and S all constant terms that independent of the length L and remain fixed when L changes, the complexity can be further reduced to (T.).
Proposition 2. The time and space complexity for the Cross-variable GNN is O(K var x D) and amounts to O(D) when kvar is constant w.r.t. D.
Proof. Without loss of generality, we assume that the number of homogeneous and heterogeneous correlated nodes for each variable are both K var, For cross-variable graph, there total of
17

---

Σn 2K = 2K" D correlated variable node pairs, Correspondingly, since the complexity of graph computation is related the number of edges, the time and space complexity of vross-variable GNN are both O K'eer x D). As Kver is a constant that is independent of D, its complexity is linear (i.e., (D))
18


Reviews:
[{"review_id": "f2qbWy01o9", "reviewer": "Reviewer_F3ix", "summary": "This paper aims to deal with the temporal fluctuations and heterogeneity between variables, caused by unexpected noise, for better multivariate time-series forecasting. Specifically, the authors propose a linear complexity CrossGNN model, including Cross-Scale GNN which captures relationships inter- and intra- scales, and Cross-Variable GNN which captures the homogeneity and heterogeneity relationships between different variables. Experiments on 8 benchmark multivariate time-series datasets demonstrate the effectiveness of CrossGNN over some existing methods.", "questions": "1. Please compare with SoTA methods, like RLinear, RMLP and PatchTST.\n2. Please give explanations on the weaknesses 2~4.", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The authors propose a GNN-based (CrossGNN) method with linear complexity for long-term time series forecasting.\n2. The CrossGNN captures the relationships between both scales and variables.\n3. The paper is easy to understand.", "weaknesses": "1. The proposed CrossGNN does not compare with the SoTA methods, like RLinear, RMLP [1] and PatchTST [2]. I checked the results in this paper (Table 1 & Figure 4) and [1,2], and found that CrossGNN is worse than RLinear, RMLP and PatchTST.\n\n[1] Revisiting Long-term Time Series Forecasting: An Investigation on Linear Mapping\n\n[2] A Time Series is Worth 64 Words: Long-term Forecasting with Transformers\n\n2. It is unclear why the heterogeneity between variables is caused by unexpected noise.\n3. In the top of Page 5, it is unclear why the production of two learnable vectors can diminish the effect of noise.\n4. What is the meaning of \"C\" in Figure 2?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer F3ix,\n\nThanks for your valuable comments for improving our manuscript. Firstly, we have incorporated a comparison with the SoTA baselines. Subsequently, we have seriously addressed and clarified the concerns  you raised as below.\n\n**W1: Lack of SoTA baselines.**\nBased on your suggestion, we are working diligently to conduct experiments and have added the abovementioned three SoTA baselines for comparision  **in Table 1 and Table 2 of the one-page PDF**. \n\nNevertheless, directly comparing  our solution with  RMLP, RLinear, PatchTST may be unfair.\nThe reasons can be two aspects,\n\n1. **Publication date.** The publication date of RLinear, RMLP (18th, May) on arxiv is after the  deadline (17th, May) of NeurIPS. \n\n2. **Different experimental settings.** Our CrossGNN takes 96 time steps as input while the input of RMLP, RLinear, PatchTST are 336, which are longer than ours. It is obvious that much longer sequence can lead to higher performance but deterioriates the efficiency.\n\n\nFinally, we emphasize that we now conduct fair comparison experiments in the following ways: \n\n - **Input length 96:**  We directly compared CrossGNN with PatchTST, RMLP, RLinear with the same input length. \n - **Input length 336:** Similar to RMLP, RLinear, and PatchTST, we also incorporated the **RevIN** [3] technique in CrossGNN to mitigate data shift effects. \n\nThe final experimental results (**in Table 1 and Table 2 of the one-page PDF**) demonstrate that our CrossGNN still outperforms recent SoTA methods, simultaneously maintaining linear complexity. \n\nThank you again for suggestions in improving our experiments. We will include the complete experimental results in the final version and cite the two referenced papers  \\[1\\]\\[2\\] accurately.\n\n[1] Revisiting Long-term Time Series Forecasting: An Investigation on Linear Mapping\n\n[2] A Time Series is Worth 64 Words: Long-term Forecasting with Transformers\n\n[3] Reversible instance normalization for accurate time-series forecasting against distribution shift. ICLR, 2022.\n\n\n\n\n**W2: Heterogeneity between variables caused by unexpected noise.**\n**Heterogeneity between variables:**  Referring to some variables exhibiting distinct temporal patterns.  In a stable physical system, most variables exhibit homogeneous correlations. Heterogeneity generally arises from unexpected noise, this can be summarized into two scenarios: \n\n**(1) Persistent influence:** Continuous noise, such as emergency events, leads to sustained changes in the temporal patterns of certain variables, resulting in disparate time series. For instance, in the Traffic dataset, some traffic intersections exhibit higher nighttime traffic than daytime due to accidents during the day.\n\n **(2) Instantaneous influence:** Brief yet high-frequency noise, like sensor distortions, introduces irregular data points into the time series of certain variables, reducing the overall sequence regularity.\n\nThe above kinds of noise can affect the time signal as two aspects, 1) different variables must exhibit various evolutionary patterns, 2)  the correlations between variables are varying along time，thus we design Cross-Variable module to effectively enhance the role of cross-variable relationship learning in time series prediction.\n\n\n\n\n\n\n\n\n\n**W3: The production of two learnable vectors.**\n\nEq(6) at the top of Page 5 aims at learning stable and cross-scale temporal correlations $E^{scale}\\in R^{L'\\times L'}$ not affected by input noise. Here, we explain why it can mitigate the impact of input noise on learning temporal correlations:\n\n- **Input-independent**: Previous works learns the correlation weight by considering the correlation as the function of input $X$ while in this paper, we initialize the learnable correlation $E^{sacle}$  as the production of two learnable vectors $vec_1^{scale}$ and $vec_2^{scale}$, which are independent of $X$. Thus, it cannot be disturbed by noisy $X$ and directly learn the correlation regularity from general data sequences.\n- **Cross-scale**: We extend $E^{scale}$  into multiple scales with coarse sequence, as shown in main text Figure 1(c), where the insight is that coarse temporal scale extracts the backbone of regularity and thus less influenced by noise.\n\nBesides, we can also verify the significant motivation of our denoising insight from the analysis of self-attention. Through extensive experiments of noise impact on learning long temporal correlations, we observed cases where abnormal noise received excessive attention, as depicted in **Figure 1 of one-page PDF**. Thus, exploring dependency structure independent of inputs noise  is crucial.\n\n\n\n**W4: The meaning of \"$C$\" in Figure 2.** $C$  refers to the (number of) channel dimension. Here, we employ an expansion dimension strategy, creating an embedding for each time step.  This strategy is inherited from MTGNN [4], aiming to enhance the local semantics at each time step and positively impact subsequent cross-scale and cross-variable interactions.  Thanks for you question, we will add the explanation in the final version.\n\n[4] Multivariate time series forecasting with graph neural networks. SIGKDD, 2020.\n\nAuthors of Paper 3421"}, {"review_id": "DxKfzusorM", "reviewer": "Reviewer_zRqD", "summary": "Overall Comment：\nThis article addresses two issues in multi-variate time-series modeling: i) How to address signal noise in multivariate time series, and ii) How to address interactions between multiple variables to extract information. The article proposes two GNN models to solve these problems, including the Cross-Scale GNN model for addressing signal noise and the Cross-Variable GNN for addressing interactions between different signals.\n\nOverall, while some parts of the article are not very clear, such as, what is the meaning of  homogeneity and heterogeneity in time series, and the the ablation study can be improved, the article's innovative ideas are clear and worth being seen by more people. Compared to conventional baseline methods, this article has significant advantages.\n\nThe strengths of the article include:\n\n1. The paper conducts a large number of experiments, tested on 8 datasets. From the experimental results, the proposed method is effective and performs better than conventional MTS baseline methods.\n2. The article is well written and easy to read, making it easy to understand the information that the author is trying to convey.\n\nThe weaknesses of the article include:\n\n1. The analysis of the ablation study needs to clarify that which module, the Cross-Scale GNN and Cross-Variable GNN, plays a larger role in the model in order to judge whether it is filtering signal noise that makes the proposed model perform better on each dataset or whether introducing interactions between multiple variables is the key factor.\n2. Correspondingly, if the Cross-Scale GNN model for addressing signal noise plays a larger role, is the focus of the article on time series denoising or modeling time series with noise? The article should be compared with methods specifically designed for denoising time series, rather than just conventional time series modeling baselines.\n3. What is the meaning of homogeneity and heterogeneity in time series signals? In Figure 1(d), only positive or negative correlations between variables are displayed, which is not related to homogeneity and heterogeneity relationships between variables.", "questions": "As I write in the Weaknesses: \n\n1. the Cross-Scale GNN and Cross-Variable GNN, which plays a larger role in the model？ Whether it is filtering signal noise that makes the proposed model perform better on each dataset or whether introducing interactions between multiple variables is the key factor?\n\n2. What is the meaning of homogeneity and heterogeneity in time series signals? In Figure 1(d), only positive or negative correlations between variables are displayed, which is not related to homogeneity and heterogeneity relationships between variables.", "limitations": "The authors didn't address the limitations clearly.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The strengths of the article include:\n\n1. The paper conducts a large number of experiments, tested on 8 datasets. From the experimental results, the proposed method is effective and performs better than conventional MTS baseline methods.\n2. The article is well written and easy to read, making it easy to understand the information that the author is trying to convey.", "weaknesses": "The weaknesses of the article include:\n\n1. The analysis of the ablation study needs to clarify that which module, the Cross-Scale GNN and Cross-Variable GNN, plays a larger role in the model in order to judge whether it is filtering signal noise that makes the proposed model perform better on each dataset or whether introducing interactions between multiple variables is the key factor.\n2. Correspondingly, if the Cross-Scale GNN model for addressing signal noise plays a larger role, is the focus of the article on time series denoising or modeling time series with noise? The article should be compared with methods specifically designed for denoising time series, rather than just conventional time series modeling baselines.\n3. What is the meaning of homogeneity and heterogeneity in time series signals? In Figure 1(d), only positive or negative correlations between variables are displayed, which is not related to homogeneity and heterogeneity relationships between variables.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer zRqD,\n\nThank you for your valuable insights  for polishing our manuscript. We have conducted additional experiments and analysis to address your concerns.\n\n**W1&Q1 Which module is more important.**\n\nThank you for your question in our experimental analysis. Our ablation experiments in manuscript section 4.4 have revealed that the Cross-Scale module holds greater significance. Across twelve settings in three datasets, the removal of Cross-Scale GNN resulted in performance degradation in all settings, positioning it at the bottom of the rankings in ten settings. This pivotal finding emphasizes the effect of Cross-Scale GNN on modeling temporal relationships, effectively filtering signal noise, and enhancing downstream prediction outcomes. In the final version, we will emphasize the importance of the Cross-Scale module in the Ablation section, acknowledging your guidance in polishing our experimental analysis.\n\n**W2: Add comparative baselines & Robust analysis on noise.**\n\nThank you for your valuable suggestions. \n\n- We have incorporated comparisons with denoising time series prediction methods, such as Stationary (a model designed for unstable time series), and presented the experimental results **in Table 1 and Table 2 of one-page PDF.** The  results demonstrate that our CrossGNN still outperforms other denosing time series prediction methods.\n\n- Additionally, to validate CrossGNN performance in time series prediction with noisy data, we introduced varying levels of Gaussian white noise to three datasets (ETT2, ETTm2, and Weather). The experimental outcomes in **the Appendix B.1** illustrate that  as the intensity of Gaussian white  noise increases, the signal-to-noise ratio (SNR) gradually decreases from 100 dB to 0 dB. As the SNR decreases from 100db to 0db, the mean square error (MSE) of CrossGNN increases more slowly than MTGNN and ETSformer. The quantitative results demonstrate that CrossGNN exhibits good robustness against noisy data and has a great advantage when dealing with unexpected fluctuations. We speculate such improvements benefit from the explicit modeling of respective scale level and variable-level interactions.\n\n**W3&Q2: Meaning of homogeneity and heterogeneity.**\n\nThank you for your advice in improving our expression. In temporal signals, heterogeneity indicates that the temporal patterns between variables are not similar, while homogeneity indicates similar temporal patterns between variables. In the manuscript,  we use negative correlation to represent heterogeneous variables and positive correlation to represent homogeneous variables.\n\nThanks for providing the encouraging reviews and your valuable suggestions indeed make sense to further improve our manuscript. Thanks again!\n\nAuthors of Paper 3421"}, {"review_id": "lJ4rtz8HAU", "reviewer": "Reviewer_MKp9", "summary": "CrossGNN is a linear complexity GNN model designed for MTS forecasting, addressing two obstacles: self-attention mechanisms assigning high scores to outlier points and real-world data homogeneity and heterogeneity. By combining Adaptive Multi-Scale Identifier (AMSI), Cross-Scale GNN, and Cross-variable GNN, CrossGNN outperforms recent SOTA methods in real-world datasets.", "questions": "In Figure 2, all scales are concatenated in time dimension from coarse to fine scale. Why this kind of concatenation is needed? Is there any reason for this?", "limitations": "There seems no limitation in this paper. Please include one.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. Despite considering the relationship between variables, unlike Transformer-based models, it has a low time complexity O(L).\n2. The paper also includes a comprehensive set of ablation studies and the performance in forecasting is SOTA.\n", "weaknesses": "1. The authors mentioned that noise can disturb attention mechanism in transformer. However, the noise ratio is less than 2\\% as shown in Figure 1 (b) and I do not think this would be significant. Also, those outliers can be removed by using outlier detection techniques before training. \n2. It is necessary to add an explanation or experiments to assure that the self-attention mechanism really adversely affects performance by assigning outlier points as high scores.\n3. Technical novelty is a bit weak. Concepts of cross-scale and cross-variable are already introduced concepts in TimesNet and Crossformer. Also, many papers have already used GNN to deal with cross-variable interaction. What is the main difference except the input?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer MKp9,\n\nThank you for your insightful advice for polishing our manuscript. We have conducted sufficient experiments and analysis to dispel your concerns. The details can be found below. \n\n**W1: Noisy illustration issue in Figure 1 (b) & Whether it works when outliers removed.**\n\n(1) This is an illustrative example in Figure 1 (b). We allow the noise increase and present the varying results as illustrated in **Appendix B.1** to verify the effectiveness of CrossGNN. It demonstrates CrossGNN exhibits good robustness against noisy data and has a great advantage when dealing with unexpected noise. \n\n\n\n(2) The significance of denoise learning and reasons for not ruling out outliers directly：\n\n- **Noise can be inevitable in real world.** Figure 1 (b) provides an illustrative example based on four academic datasets. It is possible that the noise ratio in our actual application may be much higher than the reported 2%.\n- **Current anomaly detection algorithms may not necessarily detect all types of noise.** The noise can be summarized into two scenarios: persistence influence (e.g. the traffic accidents), instantaneous influence (e.g., sensors distortion). Some noise may exhibit sudden increases or decreases, while others may be continuous, low-amplitude variations.\n- **Directly filtering out outlier noise can result in significant information loss.** If we remove all the noisy samples detected by the noise detector, the remaining available samples may not provide sufficient support for training.\n\n\nIn addition, we perform experiments by removing outlier points with noise detection algorithms [1] on four datasets. The prediction results are shown in **Table 3 of the one-page PDF**. We find that directly removing noisy samples have not improved the prediction performance.\n\n[1] Robust regression and outlier detection. 2009.\n\n**W2: Explanation and experiment of noise effect on self-attention.**\n\nThank you for highlighting the importance of validating the impact of noise on the self-attention mechanism. We  have investigated the experimental results on four datasets and provide the explanation to your concerns from both  theoretical analysis and  empirical perspectives. \n\n**Theoretical analysis:**\nIn the self-attention mechanism, for a given input time series data $X=[x_1,x_2,x_3,...,x_L]\\in R^{L\\times D}$, where $L$ denotes the sequence length and $D$ represents the embedding size, the attention score of the $i$-th time step with respect to the $j$-th time step can be expressed as $a_{i,j}=\\frac{e^{(x_i\\times q)\\cdot (x_j \\times k)}}{\\sum_{k=1}^{L}e^{(x_i\\times q)\\cdot (x_k \\times k)}}$, where $q\\in R^{D \\times C}$ and $k\\in R^{D\\times C}$ are the query and key matrices, respectively. It can be observed that the score $a_{i,j}$ depends on the value of $x_j$. When $x_j$ is a high-value outlier and the weights $q$ and $k$ are positive, $a_{i,j}$ will be assigned with a larger value. \n\n**Experimental results:**\n\n**(1) High score.** We trained predictive models using a Vallina Transformer on the ETTh1, Traffic datasets. **In Figure 1 of the one-page PDF,** the randomly sampled sequences and their corresponding attention matrices  demonstrate that applying the self-attention mechanism to time series data can indeed assign higher scores to outlier points.  \n\n**(2) Performance  degradation.**  To evaluate the robustness of CrossGNN and transformer-based models against noise, we add different intensities of Gaussian white noise to the original MTS and observe the performance changes. **These results are in Appendix B.1.**  We found that by directly reducing the signal-to-noise ratio (increasing the noise intensity), the performance of the transformer model decreased rapidly, while CrossGNN decreased more slowly. It means noise will have a greater negative impact on the prediction performance of transformer-based models.\n\n\n\n\nWe also observed in recent work [2] that visual Transformers might excessively focus on a few abnormal tokens, termed the \"token overfocusing phenomena.\" This aligns with our perspective. We will include these experimental results in the final version, thanks for your suggestion to enhance the quality and validness of our manuscript.\n\n[2] Robustifying Token Attention for Vision Transformers. ICCV,2023.\n\n\n\n**W3: Technical novelty.**\n\nThank you for your question. We have provided a detailed description of the technical novelties of CrossGNN in the global rebuttal and explained the differences between CrossGNN and other methods such as TimesNet, Crossform, and GNN-based.\n\n- **Time-interaction**.\n   \n1. **First attempt to explore GNN for temporal modeling**: Exploit GNN to connect different temporal points.\n\n2. **First attempt to propose Cross-Scale information focus**: Refine interaction learning across different temporal scales to reduce noise effects.\n\n3. **Adaptable Multi-Scale MTS Acquisition**: An  Adaptive Multi-Scale Identifier to dynamically partition scales based on sequence frequency characteristic.\n\n- **In variable interaction.**\n\n1. **Pioneering Temporal Data Heterogeneity.**\n\n2. **Low computation complexity.**\n\nThese systematical novel solutions can jointly resolve the new problem of reducing noise in time-series and contribute to noisy-reduced learning scheme. Detailed descriptions of these technical novelty and differences from other models can be found in our common responses to all reviewers.\n\n**Q1: Concatenation of Scales.**\n\nThe reasons can be three aspects. \n\n1. The direct concatenation can well preserve the full multi-scale information.\n\n2. To guarantee  cross-scale interaction, concatenating sequences of varying scales allows direct matrix multiplication for time-step level interactions between different scales.\n\n3. This concatenation leverages the neuron connections and learnable weights of Fully Connected (FC) layers to facilitate feature extraction. \n\nAuthors of Paper 3421"}, {"review_id": "i42jDKF2Bz", "reviewer": "Reviewer_LSa5", "summary": "This work first analyzes real-world datasets for multivariate time series forecasting and finds out two problems that are not well handled by previous works: 1) unexpected noise; 2) heterogeneity between variables. A GNN-based model, named CrossGNN, is proposed to fill the gap. CrossGNN consists of three components: 1)Adaptive Multi-Scale Identifier identifies potential periods with FFT and aggregates time series at different scales to construct a mult-scale view of the data; 2)Cross-Scale GNN constructs a graph among scales and uses GNN to capture cross-scale dependency; 3)Cross-Variable GNN captures homogeneous and heterogeneous dependency among variables. Correlation graphs used in Cross-Scale and Cross-Variable GNNs are restricted to be sparse, so the complexity is linear to the input length. Experiments on 8 real-world datasets demonstrate the effectiveness of the proposed model.", "questions": "- In Equation (1), amplitudes are averaged over variables. Is the average operation reasonable? What if different variables have different periodic patterns?\n- How is the ouput of AMSI (with shape $L' \\times D$) embedded into latent vectors with shape $L' \\times D \\times C$?\n- Settings of the ablation study should be described in detail, perhaps in the appendix. What does \"divides the scales directly by fixed length\" mean? Does C-Hete only remove the heterogeneous connections? If so, an ablation study that removes both homogeneous and heterogeneous connections should be conducted.\n- It is better to compare the efficiency with baselines such as DLinear, Pyraformer, and TimesNet in Figure 6, as it is obvious that the proposed model is more efficient than the full graph. Moreover, the complexity of Pyraformer is $O(L)$, not $O(L \\log L)$.\n- See my other questions in the weakness part.", "limitations": "No potential negative societal impacts.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This work is well-structured and easy to follow.\n- The studied problem, noise in datasets, is an important and practical problem for MTS forecasting.\n- Three components in CrossGNN are well-motivated and reasonable. The linear complexity is also an advantage.\n- Robustness analysis of noise in section 4.3 is interesting.", "weaknesses": "- My main concern is that graphs used in Cross-Scale and Cross-Variable GNNs are static for each dataset. These graphs are constructed by shared learnable vectors with filtering, so they are static and the same for different inputs. This leads to 1) the dependency structure of time steps being fully determined by the positions in the series; 2) the dependency structure of variables being the same for different inputs in a dataset. \n- Some recent works (e.g. PatchTST[1], Crossformer[2]) that explicitly model the cross-variable dependency should be compared in the main experiment.\n\n[1]Yuqi Nie, Nam H. Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. In International Conference on Learning Representations, 2023.\n\n[2]Yunhao Zhang and Junchi Yan. Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting. In International Conference on Learning Representations, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer LSa5, \n\nThanks for your providing a positive feedback to our manuscript and encourage us to make further improvements. Now, we have addressed your concerns by supplementing both experimental studies and concise technical descriptions.\n\n**W1: Fixed Graph Structure for Different Inputs.**\nBased on your suggestions, we have analyzed the effects of a fixed dependency structure in CrossGNN from both temporal dimensions and variable dimension.\n\n- **Fixed Graph Structure in Temporal Dimension：**\nIn a stable physical system, the relationship between time points is usually consistent, exhibiting a fixed change pattern over time. Such consistency can be influenced by noise, inducing the graph signal to be time-varying and dynamic with less regularity. Therefore, we are expected to capture the general and consistent relationship via GNN and learn a fixed temporal graph structure to avoid the impact of such noise. \n\n- **Fixed Graph Structure  in Variable Dimension：**\n  (1) A fixed dependency structure can effectively represent variable relationships across diverse inputs (remain robust under different input disturbance). And such effectiveness has been validated in MTGNN. (2) Our Cross-Variable GNN further decomposes variable-level relationships into positive and negative ones, enabling a more comprehensive and effective description of dependencies.   (3) Fixed graph structure can reduce forward computation time.\n\nWe aim to learn a stable relationship between different time steps and variables within a fixed physical system (each dataset), reducing noisy effects on dependency structure. The concern you raised is the exact one we have considered in our work, yet the experimental results of CrossGNN can indicate that a fixed correlation matrix tends to be more effective compared to the variable correlation matrix. We appreciate your thoughtful analysis and raising this concern, we will place above analysis at the conclusion of our manuscript.\n\n**W2: Additional Comparative Experiments**\n\nThank you for pointing out the experimental limitations. We worked day and night, conducting numerous experiments, and have documented the complete results in the **Table 1  and Table 2 of our submitted one-page PDF**. \n\nExperiments demonstrate that our method continues to exhibit excellent predictive capabilities compared to PatchTST and Crossformer. We appreciate your assistance and suggestions in enhancing the persuasiveness of our experiments, and we will include the complete results in the final version of our manuscript.\n\n**Q1: Averaged amplitudes.**\nThere are two common strategies to achieve multi-scale time-series.\n\n1. **As you mentioned**, each variable derives a variable-wise period based on its own frequency domain amplitudes. Then, each variable obtains distinct period length and subsequently forms variable-wise multi-scale MTS.\n\n2. **In CrossGNN**, we average the frequency domain amplitudes of each variable and uniformly assign the same periods to all the variables, resulting in our multi-scale MTS.\n\nWe take the latter strategy for following two reasons:\n\n1. As most variables exhibit similar temporal patterns, the first strategy leads to less distinguishable amplitudes among variables.\n\n2. Given comparable performance, the latter approach is more efficient in implementation, as it allows for parallel operations on all variables. We provide runtime and corresponding prediction metrics for the above two strategies in **Table 6 of the one-page PDF**  for your reference. The results demonstrate that averaging the amplitudes in CrossGNN achieves  comparable prediction performance while significantly reduces the running time.\n\n\n\n**Q2: How is $L^{\\prime}\\times D \\rightarrow \\ L^{\\prime}\\times D\\times C$.**\n\n $C$  refers to the (number of) channel dimension. Here, we employ an expansion dimension strategy (using an MLP), to create an embedding for each time step.  This strategy is inherited from MTGNN, aiming to enhance the local semantics at each time step and positively impact subsequent cross-scale and cross-variable interactions.  Thanks for you question, we will add the explanation in the final version.\n\n\n\n**Q3: Ablation Study.**\n\nThank you for your valuable suggestion regarding ablations.\n\n1. In AMSI, we learn $k$ potential periods of the sequences and perform average pooling based on these $k$ periods to obtain different scale representations. In **C-AMSI**, we remove the adaptive learning of period lengths and directly use $k$ fixed lengths (e.g., $1, 2, 3, ..., k$) for average pooling.\n2. Thanks to the  insights of reviewers, we add a complete ablation study. **C-CV**: removing both homogeneous and heterogeneous connections. The results are provided in **Table 4 of the one-page PDF.** The results demonstrates that removing the entire Cross-Variable GNN module results in a significant performance degradation.\n\nWe appreciate your suggestions for expression clarity and additional ablation experiments. We will incorporate these revisions  and additional results to our final version.\n\n**Q4: Training Efficiency & Complexity of Pyraformer.**\n1. We have re-evaluated the time complexity of Pyraformer and confirmed it to be $O(L)$. Thanks for pointing it out and we will make the corrections in the corresponding section of the article.\n2. Thank you for helping us enhance the validness of our paper. We comprehensively compare the per-batch running time of our model with well-acknowledged deep forecasting models (TimesNet, DLinear, Pyraformer). The results are conducted with the official model configuration, and are recorded in **Table 5 of the one-page PDF.** The results demonstrate CrossGNN the computation time of CrossGNN is significantly better than other models except for DLinear.\n\nThank you again for your insightful suggestions and reviews.\n\nAuthors of Paper 3421"}]

Author rebuttal:
Dear Reviewers,

We would like to thank you for your valuable time and constructive comments on our manuscript and we have made sufficient improvements of our work according to your comments. Here we list the major improvements below.

- **Enhanced Clarifications**: We have enhanced our manuscript with more in-depth explanations where necessary, such as an explanation of ablation studies.

- **Additional Experiments**: Based on the feedback, we have conducted additional experiments to further validate the effectiveness of our proposed model. These include but not limited a comparison with recent SoTA method such as RLinear, PatchTST, Crossformer, a comparison of running time  among well-acknowledged deep forecasting models.

- **Clearer Definitions**: We have added clear definitions for terms like 'heterogeneity between variables' to ensure readers have a comprehensive understanding of our method.


**Technical novelty of CrossGNN.**

The contribution of CrossGNN extends highlighting the challenges of unexpected noise and variable-wise heterogeneity. It also introduces significant innovations in enhancing interactions across time and variable dimensions. Next, we will elaborate our technical novelty and  highlight the distinctions between CrossGNN and other SoTA methods. 

- **In time interaction.** 

  1. **First attempt to  explore GNN for temporal modeling.**

    To the best of our knowledge, this is the first attempt to explore GNN for capturing temporal  relationships, which is beyond capturing variable-level correlation in  previous literature. We elucidate how to exploit GNN to connect different temporal points to  learn a stable and general temporal graph, which can be  free from noise interference.

  2. **First attempt to propose Cross-Scale interaction.**

  We  emphasize the significance of point-level temporal interactions across scales.  Based on the insight that coarse temporal scale extracts the backbone of regularity, we introduce the cross-scale module, which directly extracts  coarser-scale information and extends the temporal learning towards multiple scales. CrossGNN allows  refined interaction learning across different temporal scales, resulting in noise-reduced effects. 

  3. **Adaptable multi-scale MTS acquisition.**

    We devised an Adaptive Multi-Scale Identifier (AMSI), capable of dynamically partitioning scales based on input sequence frequency characteristics. This distinguishes our approach from prior methods that rely on fixed-length manual scaling, contributing to an automatic multi-scale sequence acquisition.

- **In variable interaction.**

  1. **Pioneering temporal data heterogeneity.**

     We first  propose Cross-Variable learning via decoupling temporal homogeneous-heterogeneous relationships.

  2. **Low computation complexity.**

     We maintain the computational complexity of both the Cross-scale and Cross-variable GNN modules at a linear level by constraining the number of inactive edges. 

In summary, we outline the distinctions and innovations that set us apart from the similar methods the reviewer mentioned:

- **Differences from TimesNet.**  Even TimesNet  extends 1-D  series data to a 2-D space, it still lacks explicit modeling interactions across different  temporal scales.

- **Differences from Crossformer.** Crossformer solely focuses on homogenous associations and ignores heterogeneity in the real world. Besides, the time complexity of Crossformer is $O(rDL)$ ($r$ for router vectors, $D$ for variable count, $L$ for sequence length), while CrossGNN is $O(D)$. 

- **Differences from GNN-based Approaches.** GNN-based time series prediction methods merely  capture cross-variable relationships. However, they overlook (1) heterogeneity among variables, (2) complexity of $O(N^2)$, without graph pruning for efficiency, (3) the potential of GNN-based approaches in capturing temporal relationships, fail to exploit GNN to model series along temporal dimensions.

We will polish the  technical novelty in our manuscript based on above clarification.  


Finally, we believe that these revisions have significantly improved our manuscript. We hope that our responses and the changes made address the concerns of the reviewers adequately.

Once again, thank you for your time and effort in reviewing our work. We look forward to your continued feedback.

Best regards,

Authors of Paper 3421


Meta-review: The authors propose a GNN-based method for time series modeling to captures  temporal fluctuations and heterogeneity between different variables and to model multi-scale time series. The reviewers praised the clarity of the presentation, the efficiency of the method, its comprehensive ablation studies and high performance, especially considering the low time complexity. During the discussion phase, the authors addressed the main concerns raised by the reviewers. Following the discussion, the reviewers unanimously accepted the paper. Due to the importance of time series analysis, the community would benefit from the publication of the paper and the availability of the code for this method, so I recommend acceptance.
Decision: Accept (poster)