Title: CrossGNN: Confronting Noisy Multivariate Time Series Via Cross Interaction Refinement
Venue: NeurIPS 2023
Accepted: yes
Keywords: Time Series Forecasting;
Abstract: Recently, multivariate time series (MTS) forecasting techniques have seen rapid development and widespread applications across various fields. Transformer-based and GNN-based methods have shown promising potential due to their strong ability to model interaction of time and variables. However, by conducting a comprehensive analysis of the real-world data, we observe that the temporal fluctuations and heterogeneity between variables are not well handled by existing methods. To address the above issues, we propose CrossGNN, a linear complexity GNN model to refine the cross-scale and cross-variable interaction for MTS. To deal with the unexpected noise in time dimension, an adaptive multi-scale identifier (AMSI) is leveraged to construct multi-scale time series with reduced noise. A Cross-Scale GNN is proposed to extract the scales with clearer trend and weaker noise. Cross-Variable GNN is proposed to utilize the homogeneity and heterogeneity between different variables. By simultaneously focusing on edges with higher saliency scores and constraining those edges with lower scores, the time and space complexity (i.e., $O(L)$) of CrossGNN can be linear with the input sequence length $L$. Extensive experimental results on 8 real-world MTS datasets demonstrate the effectiveness of CrossGNN compared with state-of-the-art methods.

CrossGNN: Confronting Noisy Multivariate Time Series Via Cross Interaction Refinement

Abstract

Recently, multivariate time series (MTS) forecasting techniques have seen rapid development and widespread applications across various fields. Transformer-based and GNN-based methods have shown promising potential due to their strong ability to model interaction of time and variables. However, by conducting a comprehensive analysis of the real-world data, we observe that the temporal fluctuations and heterogeneity between variables are not well handled by existing methods. To address the above issues, we propose CrossGNN, a linear complexity GNN model to refine the cross-scale and cross-variable interaction for MTS. To deal with the unexpected noise in time dimension, an adaptive multi-scale identifier (AMSI) is leveraged to construct multi-scale time series with reduced noise. A Cross-Scale GNN is proposed to extract the scales with clearer trend and weaker noise. Cross-Variable GNN is proposed to utilize the homogeneity and heterogeneity between different variables. By simultaneously focusing on edges with higher saliency scores and constraining those edges with lower scores, the time and space complexity (i.e., O ( L ) ) of CrossGNN can be linear with the input sequence length L . Extensive experimental results on 8 real-world MTS datasets demonstrate the effectiveness of CrossGNN compared with state-of-the-art methods. The code is available at https://github.com/hqh0728/CrossGNN.

1 Introduction

Time series forecasting has been widely used in many fields (i.e., climate [1], traffic [31], energy [3], finance [13], etc) [8, 29, 20, 10]. Multivariate time series (MTS) consists of time series with multiple variables and MTS forecasting aims at predicting future values based on historical time series. Deep learning models [26, 24, 17, 4, 21] have demonstrated superior performance in MTS forecasting. In particular, Transformer-based models [34, 27, 35] have achieved great power in MTS, benefiting from its attention mechanism which can model the long-term interaction between different time points of sequences (cross-time). Graph Neural Networks (GNNs) [28, 16, 2, 19, 7] have also shown promising results for MTS forecasting, which can extract pre-defined or adaptive interaction between different variables (cross-variable).

However, a recent study [33] shows that a simple linear model dramatically outperformed many stateof-the-art (SOTA) models, and it inspires us to investigate the reasons why the existing cross-time

This work is done when Qihe Huang was an intern at YouTu Lab.

/a0 Yang Wang and Zhengyang Zhou are corresponding authors.

and cross-variable interaction modeling fail to enhance prediction performance. By conducting a thorough analysis on real-world data, we observe that the presence of some unexpected noise (caused by humans, sensor distortion) may be responsible for it. In time dimension , as shown in Figure 1 (a), Transformer-based model heavily relies on the input sequences to generate the attention map, while its prediction may be susceptible to incidental noise, even some small fluctuation (i.e., noise) could easily lead to significant shifts on temporal dependencies. Our findings reveal that self-attention mechanism tend to assign high scores to outlier points in time series, resulting in spurious cross-time correlations. In variable dimension , cross-variable correlation exhibits a complex and dynamic evolution over time [18]. Despite the existence of underlying causal associations between variables, extracting cross-variable interaction is difficult due to the influence of noise interference. Additionally, as shown in Figure1 (b), we observe that such unexpected noise, which can be detected by the outlier detection algorithm [14], accounts for a high proportion in the time series.

Unexpected

noise

Transformer-based prediction

Input data

Ground truth

3.00%

-96057

Seale 3

Seale 1

Seale 2

Reduce sudden noise

Fine

Variable 1

Variable 2

Variable 3

→Negatively correlated

7

2.00%

High attention

score

1.50%

1.00%

Pooling

SLSM

-9050

I/

1/

0.00%

Pooling

Coarse

Positively correlated

V

ETTh1

ETTm1

Traffic

Weather

(a)

(b)

(c)

(d)

Despite the non-negligible noise in time series, we can still discover the potential opportunities to confront noise challenges. (1) Cross-Scale Interaction. As shown on Figure 1 (c), by performing multi-scale extraction on the time series, we observe that different scales possess distinct levels of noise intensity, typically with coarser scales exhibiting lower noise intensity. Evidently, capturing dependencies across different scales enables the cross-time relationships to be robust against noise [15]. (2) Cross-Variable Interaction. As shown on Figure 1 (d), there is both homogeneity and heterogeneity for cross-variable interaction in real-world data [36]. In fact, the two relationships can both contribute to invariant connections during the temporal progression. Consequently, learning the invariant associations that contain both homogeneous and heterogeneous relationships among variables can boost its robustness to confront noise. Based on above analysis, it is still challenging to refine interaction in noisy MTS. The key obstacles can be summarized as follows: 1) How to capture cross-scale interaction that is not sensitive to unexpected input noise. 2) How to extract cross-variable relations between heterogeneous variables.

In this work, we propose CrossGNN, which is the first GNN solution to refine both cross-time and cross-variable interactions for MTS forecasting. To deal with the unexpected noise in time dimension, we firstly devise an adaptive multi-scale identifier (AMSI) to construct multi-scale time series with different noise levels. In time dimension, we propose Cross-Scale GNN, which is a temporal correlation graph, to model the dependency between different scales. The scales with clearer trends and weaker noise will be assigned with more edge weight. In variable dimension, we first introduce heterogeneous interaction modeling between variables into MTS forecasting and propose cross-variable GNN to utilize the homogeneity and heterogeneity between different variables with positive and negative edge weights. By focusing on edges with higher saliency scores and constraining those edges with lower scores at the same time, CrossGNN achieves linear time and space complexity (i.e., O(L) ) with input sequence length L . The main contributions are summarized as follows:

We conduct comprehensive studies on real-world MTS data and discover that the unexpected noise in time dimension and variable-wise heterogeneity between variables is not well handled by existing Transformer-based and GNN-based models.

We propose a linear complexity CrossGNN model, which is the first GNN model to refine both cross-scale and cross-variable interaction for MTS forecasting.

1) To deal with the unexpected noise in time dimension, AMSI is leveraged to construct multi-scale time series with different noise level and a Cross-Scale GNN is proposed to capture the scales with clearer trend and weaker noise.

2) Cross-Variable GNN is designed to model the dynamic correlations between different variables. This the first model to introduce heterogeneous interaction modeling between variables into MTS forecasting.

Extensive evaluation on 8 real-world MTS datasets demonstrates the effectiveness of CrossGNN. Specifically, CrossGNN achieved top-1 performance on 47 settings and top-2 on 9 settings when compared with 9 state-of-the-art models with varying prediction lengths.

2 Related Work

Multivariate Time Series Forecasting. MTS forecasting has witnessed significant advancements due to the emergence of deep neural networks. These networks can be based on Convolutional Neural Network (CNN) [26, 23], Recurrent Neural Network (RNN) [5, 6], Transformer [11, 25, 27, 35, 9], or Graph Neural Network (GNN) [28, 32]. Generally, the primary emphasis of these studies lies in devising interactions between the temporal dimensions (cross-time) and the variable dimensions (cross-variable).

Cross-Time Interaction Modeling. Cross-time interaction modeling aims to capture correlations between different time points. Recently, CNN-based model TimesNet [26] transforms the time series into a two-dimensional matrix and uses a CNN-based backbone for feature extraction. RNNbased model LSTnet [6] utilizes the Long Short-Term Memory (LSTM) to model the temporal dependencies, but it may be limited by the inherent issue of gradient vanishing/exploding in RNNs. Transformer-based models benefit from its self-attention mechanism, enabling them to capture longterm cross-time dependency. AutoFormer [27] incorporates a decomposition mechanism that splits the input sequence into trend and seasonality, and integrate an auto-correlation module into the transformer to capture long-term cross-time dependency. FedFormer [35] leverages a frequencyenhanced decomposition mechanism while incorporating additional frequency information. However, despite the outstanding performance of Transformer-based methods, we observe that their selfattention mechanism is susceptible to unexpected noise, as shown on Figure 1(a). Based on these findings, we propose an innovative GNN-based method that constructs a cross-scale temporal graph to mitigate the impact of temporal noise on modeling cross-time correlations.

Cross-Variable Interaction Modeling. Cross-Variable Interaction is proved to be critical for MTS forecasting [34], and numerous works have employed Graph Neural Networks (GNNs) [22, 37, 30, 38] to capture cross-variable relationships. STGCN [32] firstly utilizes GNN to model the cross-variable dependency in traffic forecasting, which can effectively capture the dependency between different roads in pre-defined topology graphs. MTGNN [28] expands the utilization of GNN from spatiotemporal prediction to MTS forecasting, and proposes a straightforward method for computing adaptive cross-variable graph. On the other hand, the Transformer-based MTS prediction works have also recognized the potential of cross-variable interactions to enhance prediction performance, such as CrossFormer [34]. However, the cross-variable relationship is dynamic and can be greatly influenced by noise during the learning process. Given this, we refine the cross-variable relationship by decoupling homogeneity and heterogeneity in MTS, resulting in a noise-insensitive relationship during the temporal evolution.

3 Methodology

In long-term multivariate time series (MTS) forecasting, the input comprises historical sequences across D variables denoted by X = { X t 1 , ..., X t D } L t =1 ∈ R L × D , where L denotes the look-back window size and X t i denotes time series of the i th variate at the t th time step. The objective of MTS forecasting is to predict future time series denoted by ˆ X = { ˆ X t 1 , ..., ˆ X t D } L + T t = L +1 ∈ R T × D based on X , where T represents the prediction time steps and T ≫ 1 . The detailed structure of CrossGNN is illustrated in Figure 2. We firstly employ an adaptive multi-scale identifier (AMSI) to generate multi-scale time series and reduce noise on coarse scale. Then, we construct scale-sensitive and trendaware temporal graph to extract cross-scale interaction. We perform cross-variable aggregation via

OutputMulti-scaleMTSX'eREo

Different variable

Cross-scale interaction

Output HRDC

Channel dimension linear project

Coarse-

Fine

Concatenate all scales in time dimension

Cross-scale graph

↑

Fine

Scale Capture by AvgPool()

Coarse

6a

Scale-sensitiverestriction

Trend-aware selection

Time node

stride

Masked correlation

Long

Learnablevector

Inializedgraph

Associated correlation

period

Fine

Mark different period in

Time series at

Cross-variable interaction,

timedomain

different scales

Differe

Input HR

Output HRLC

Time dimension linear project )

Period Capture by FFT

ent

Top-S Highest frequency:

Potential period length:

f,ffsfs

varial

ble

Cross-variablegraph'

A∈R

Average amplitude acrossDvariables

Homogeneous and heterogeneous

Input MTSxeRt0

relationship disentanglement

Variable node

Masked correlation

Homogeneous correlation

Learnable vector

Inializedgraph

Heterogeneous correlation

(a) Adaptive multi scale identifier

(b) Cross interaction layer

(c) Forecasting

modeling homogeneous and heterogeneous relationships between variables. Finally, direct multi-step (DMS) forecasting is adopted in decoder to predict future time series.

3.1 Adaptive Multi-Scale Identifier

The adaptive multi-scale identifier (AMSI) is designed to capture the different scales of MTS from coarse to fine, and reduce the unexpected noise on the coarse scale. Technically, we utilize Fast Fourier Transform (FFT) to analyze the time series in the frequency domain and calculate the potential periods of the time series, inspired by [26]. We compute the amplitude of each time series at various frequencies using FFT and subsequently average the amplitude of X across variable dimension:

where Amp( · ) is the calculation of amplitude, FFT( · ) is the calculation of FFT, A ∈ R L represents the calculated amplitude of each frequency, which is averaged from D variables by Avg( · ) . We choose the frequencies { f 1 , f 2 , · · · , f S } which correspond to the Top-S amplitude values:

where arg Top-S( · ) picks out the S frequency values with the highest amplitude from A . The period lengths { p 1 , p 2 , · · · , p S } are calculated by the selected frequencies as follows:

Then, AvgPool( · ) with kernel size p s and stride p s , s ∈ { 1 , 2 , ..., S } , are applied to the MTS X ∈ R L × D in the time dimension to capture the MTS at s -th scales:

where AvgPool( · ) downsamples time series to obtain X s ∈ R L ( s ) × D , L ( s ) = ⌊ L p s ⌋ is the time series length in the s -th scale, ⌊·⌋ is the operation of rounding down. We concatenate the captured different

scales in the time dimension and obtain a period-wise multi-scale MTS X ′ ∈ R L ′ × D as the output of AMSI, L ′ = ∑ S s =1 L ( s ) is the sum of lengths across all scales:

Here, we employ an expansion dimension strategy (using an MLP), to create an embedding for each time step at different scale. This strategy is inherited from MTGNN [28], aiming to enhance the local semantics at each time step and positively impact subsequent cross-scale and cross-variable interactions. The shape of X ′ is finally expanded as R L × D × C .

3.2 Cross-Scale GNN

The Cross-Scale GNN is designed to utilize the interaction of the multi-scale MTS X ′ and extract the scales with clearer association and weaker noise. The cross-scale graph in the time dimension is represented as G scale = ( V scale , E scale ) . V scale = { v scale 1 , v scale 2 , ..., v scale L ′ } is the time nodes set in all time scales, where v scale i is the i -th time node. E scale ∈ R L ′ × L ′ assigns the correlation weights between each time node, each element in E scale meaning the correlation weight between two time nodes (inter-scale or intra-scale). The key purpose of Cross-scale GNN is to learn the cross-scale temporal correlation weights E scale that is insensitive to noise interference. To diminish the effect of noise on correlation weights, we maintain the independence by initializing E scale with the production of two learnable vectors vec scale 1 and vec sclae 2 , ensuring E scale not affected by noise hidden in input:

where ReLU( · ) is the active function regularizing the weight matrix so that each element is positive, and Softmax( · ) is the operation to ensure the weights of all nodes correlated to a particular time node sum to 1.

Scale-sensitive Restriction. We consider that for any time node, the number of its correlated time nodes on the fine scale should be more than the number on the coarse scale. For any time node v scale i , the number of its correlated nodes at s -th scale is restricted to k s = ⌈ K p s ⌉ , where p s is the period length of s -th scale, ⌈·⌉ denotes the ceiling function, K is a constant. This ensures that finer-scale time series contribute more temporal node associations. The neighbor time node set at s -th scale (i.e. the correlated time nodes at s -th scale ) of v scale i is denoted as:

where arg Top-k s ( · ) is an operation to extract k s nodes with highest correlation weight, E scale s ( v scale i ) ∈ R L ( s ) is the correlation weight of time node v scale i at s -th scale. In this way, the number of neighboring nodes at different scales can be restricted based on the correlation weight matrix E scale .

Trend-aware Selection. To ensure the temporal trends can be captured, we preserve the associations between a time node and its both preceding and succeeding nodes. Denote N trend ( v scale i ) as the trend neighbor set of time node v scale i , which can be defined as follows:

where scale ( · ) provides the scale of a time node. The trend neighbor set of v scale i consists of its adjacent time nodes (i.e., | i -j | ≤ 1 ) which share the same scale. This gives the ability to preserve temporal trends in the cross-scale correlation graph.

Correlation Weight Re-normalization. Denote N ( v i ) = N scale ( v i ) ∪N trend ( v i ) as the selected neighbor set of time nodes v scale i . The final correlation weight is re-normalized as follows:

where E scale [ i, j ] is the correlation weight between v scale i and v scale j . This step filters out nonsignificant correlation and preserves a restricted set of neighboring nodes N ( v scale i ) to each node. Also, re-normalization is applied to the retained correlation. Then, cross-scale correlation graph between time nodes at different scales is constructed.

Cross-scale interaction. After obtaining the cross-scale temporal correlation graph, we do crossscale interaction in the time dimension based on GNN, and the information propagation process will be stacked for N layers:

where σ is the activation function; W is the learnable matrix; H time is the time node feature; HN time is the aggregation of neighbor time node feature. HN time,N i, : aggregates the neighboring node feature from the previous layer correlated to the time nodes of v scale i . Then H time,N i, : is updated by the aggregated time node feature HN time,N i, : and its previous layer feature H time,N -1 i, : . Finally, the normalized H time,N : , : at N -th layer is the output of Cross-Scale GNN.

3.3 Cross-Variable GNN

Cross-Variable GNN is designed to extract the invariant correlation consisting of homogeneous and heterogeneous relationship. In variable dimension, the cross-variable graph is represented as G var = ( V var , E var ) . V var = { v var 1 , v var 2 , ..., v var D } is the variable node set, where D is the number of variables, and v var i is the i -th variable node. E var is the variable correlation weight matrix, each element in E var meaning the correlation weight between two variables. E var is initialized by production of two latent vectors vec var 1 and vec var 2 :

Heterogeneity Disentanglement. Specifically, we select the nodes with K var + highest correlation weight as positive neighbors with homogeneous connections, and take the nodes with K var -lowest correlation score as negative neighbors with heterogeneous connections. Denote E var ( v var i ) as the correlation weight of D variable nodes related to v var i . For variable v i , its two decoupled neighbor sets can be represented as N var -( v i ) = Bottom-K var -( E var ( v i )) and N var + ( v i ) = Top-K var + ( E var ( v i )) , respectively.

Correlation Weight Re-normalization. The corresponding homogeneous and heterogeneous correlation weights are derived as follows:

This process filters out edges other than homogeneous and heterogeneous edges. The weights of homogeneous edges are positively correlated with their correlation scores, while the weights of heterogeneous edges are negatively correlated with their correlation scores. Additionally, separate renormalization is applied to the weights of homogeneous edges and heterogeneous edges, respectively. Then, the cross-variable graph is constructed with disentangled homogeneous and heterogeneous correlations.

Cross-variable Interaction. For variable v i , the disentangled cross-variable message passing can be formulated as:

H var is the variable node feature; HN var is the aggregation of neighbor variable node feature. HN var,N : ,i aggregates both homogeneous and heterogeneous neighbor node feature of previous layer correlated to v var i . H var,N : ,i is updated by the aggregated variable feature HN var,N : ,i and its previous layer feature H var,N -1 : ,i . Finally, the normalized H var,N : ,i at N -th layer is the output of Cross-variable GNN.

3.4 Direct Multi-step forecasting

After obtaining the output features of Cross-Variable GNN, we exploit the direct multi-step (DMS) forecasting [33] for the decoder to predict the multi-step MTS at once. We take two MLPs as the decoder, where the first MLP C maps the time dimension of features from C to 1, while the second MLP T maps the time dimension from the historical input sequence L ′ to the output sequence length. The final prediction can be obtained by:

4 Experiment

4.1 Datasets And Experiment Setup

Datasets We conduct extensive experiments on 8 real-world datasets following [27], including Weather , Traffic , Exchange Rate , Electricty and 4 ETT datasets(ETTh1, ETTh2, ETTm1 and ETTm2). We follow the standard protocol in [27] and split datasets into training, validation and test set by the ratio of 6:2:2 for the last 4 ETT datasets, and 7:1:2 for the other datasets.

Baselines and Setup We compare our method with 9 state-of-the-art methods, including TimesNet [26]; 6 Transformer-based methods: PatchTST [12], Crossformer [34], ETSformer [25], FEDformer [35], Pyraformer [11], Autoformer [27]; GNN-based method: MTGNN [28]; simple yet powerful linear model Dlinear [33]. All the models are following the same experimental setup with prediction length T ∈ { 96 , 192 , 336 , 720 } for all datasets as in the original papers. We collect all baseline results from [26] except MTGNN, with the default look-back window L = 96 . We

reproduced the results of MTGNN [28] with look-back window L = 96 on all datasets according to the settings in the original paper. We calculate the Mean Square Error(MSE) and Mean Absolute Error(MAE) of MTS forecasting as metrics. More details about datasets, baselines, implementation, hyper-parameters are shown in Appendix A.3.

4.2 Main Results

The quantitative results of MTS forecasting using different methods is shown in Table 1. CrossGNN achieves outstanding performance on most datasets across various prediction length settings, obtaining 47 first-place and 9 second-place rankings in total 64 settings. Quantitatively, compared with the best results that Transformer-based methods can offer, CrossGNN achieves an overall 10 . 43% reduction on MSE and 10 . 11% reduction on MAE. Compared with GNN-based method MTGNN, CrossGNN achieves a more significant reduction 22 . 57% on MSE and 25 . 74% on MAE. Compared with other strong baselines like TimesNet and Dlinear, CrossGNN can still outperform them in general. Our method dose not achieve the best performance on the Electricity dataset. Further analysis reveals that the more severe out-of-distribution (OOD) problem in Electricity dataset results in a lower generalization of the learned temporal graph relations on the test set.

4.3 Robustness Analysis of Noise

To evaluate the model robustness against noise, we add different intensities of Gaussian white noise to the original MTS and observe the performance changes of different methods.

Figure 3 shows the MSE results of CrossGNN,

ETSformer [25] and MTGNN [28] under different noise ratio on ETTm2 dataset, and the input length is set as 96. As the signal-to-noise ratio (SNR) decreases from 100db to 0db, the mean square error (MSE) increases more slowly on CrossGNN (0.177) compared to ETSformer (0.191) and MTGNN (0.205). The quantitative results demonstrate that CrossGNN exhibits good robustness against noisy data and has a great advantage when dealing with unexpected fluctuations. We speculate such improvements benefit from the explicit modeling of respective scale-level and variable-level interactions.

4.4 Ablation Study

We conduct ablation studies by removing corre- sponding modules from CrossGNN on three datasets. C-AMSI removes the adaptive multi-scale identifier (AMSI) and directly use k fixed lengths (e.g., 1 , 2 , 3 , ..., k ) for average pooling. C-CS removes the Cross-Scale GNN module. C-Hete removes the heterogeneous connections and focuses on the homogeneous correlation modeling between different variables. C-CV removes both homogeneous and heterogeneous connections. We analyze the results shown in Table 2. Obs.1) Removing

0.6

CrossGNN

ETSformer

0.5

MTGNN

MSE

0.4

0.3

0.2

101

102

Cross-Scale GNN results in the most significant decrease in prediction metrics, emphasizing its strong ability in modeling the interaction between different scales and time points. Obs.2) Cross-Variable GNN also improves the model performance a lot, demonstrating the importance of modeling the complex and dynamic interaction between different variables. Obs.3) AMSI constantly improves the forecasting accuracy, suggesting that different scales of MTS contain rich interaction information.

4.5 Hyper-Parameter sensitivity

Look-Back Window Size Figure 4 shows the MSE results of models with different look-back window sizes on four datasets. As the window size increases, the performance of Transformer-based models fluctuates while CrossGNN constantly improves. This indicates that the attention mechanism of Transformer-based methods may focus much more on the temporal noise but our method can better extract the relationships between different time nodes through Cross-Scale GNN.

Number of Scales We vary the number of scales from 4 to 8 and report the MSE and MAE results on Weather and Traffic dataset. As shown in Figure 5 (a) and Figure 5 (b), We observe that the performance improvement becomes less significant after a certain number of scales, indicating that a certain scale size is sufficient to eliminate most of the effects of temporal noise. Number of Node Neighbors The number of neighboring nodes limited to each time node is mainly determined by the hyperparameter K . As shown on Figure 5 (c) and Figure 5 (d), we experiment with K values of 10, 15, 20, 25, and 30 and found that CrossGNN is not sensitive to the number of K . This indicates that it is only necessary to focus on strongly correlated nodes for effective information aggregation in temporal interaction.

ETTh2 T=336

ETTm2T=336

Traffic T=336

Weather T=336

0.550

0.500

0.525

0.38

0.36

I

I

1

1

0.65

09°0

0.36

0.34

I

I

I

1

MSE

0.475

S

0.34

MSE

0.55

MSE

0.32

0.450

0.32

0.30

0.425

0.400

0.375

1

I

I

I

I

1

1

1

0.30

0.28

0.26

1

I

I

1

1

一

1

0.50

0.45

0.40

I

I

I

I

I

1

I

I

1

0.28

0.26

1

48

96

144

192

336

504

720

48

96

144

192

336

504

720

48

96

144

192

336

504

720

48

96

144

192

336

504

720

L

L

L

L

CrossGNN

Dlinear

Autoformer

FEDformer

ETSformer

MSE (Left Y-Axis)

MAE (Right Y-Axis)

MSE (LeftY-Axis)

MAE (Right Y-Axis)

0.58

Traffic

0

Weather

0.36

Traffic

0.3250.165

Weather

0.23

0.578

0.576

0.576

0.325

0.3

0.35

0.574

0.32

0.16

0.225

0.574

0.28

0.34

0.572

0.155

0.22

0.572

0.57

0.315

0.57

0.315

0.26

0.33

0.15

0.215

0.568

0.568

0.31

0.566

0.31

0.24

0.32

0.566

0.145

0.21

95°0

0.305

0.22

0.31

0.564

0.305

0.14

0.205

0.562

0.562

0.56

0.3

0.2

0.3

0.56

0.3

0.135

0.2

5

(a)

S

6

8

5

(b)

S

6

10

15

(?)

20

K

25

30

10

15

(d)

20

K

25

30

4.6 Complexity Analysis

Table 3 illustrates the theoretical complexity of CrossGNN and existing Transformer-based methods. Detailed complexity derivation can be found in the Appendix C. To verify that the time and space complexity of our method is indeed O ( L ) , we use TVM to implement the GNN computation part and compare the computation time and memory usage of with full connected graph during inference on ETTh2. Comparison experiments are implemented on a Intel(R) 8255C CPU @ 2.50GHZ with 40GB memory, centos 7.8, and TVM 1.0.0. Figure 6 illustrates the time and memory cost of GNN modules, and our proposed approach is close to linear with the input length.

5 Conclusion and Future Work

In this work, we construct a comprehensive analysis of real-world data and observe that the temporal fluctuations and heterogeneity between variables, caused by unexpected noise, are not well handled by current popular time-series forecasting methods. To address above issues, we propose a linear complexity CrossGNN model, which is the first GNN model to refine both cross-scale and crossvariable interaction for MTS forecasting. An adaptive multi-scale identifier (AMSI) is leveraged to obtain multi-scale time series with reduced noise from input MTS. In particular, Cross-Scale GNN captures the scales with clearer trend and weaker noise, while Cross-Variable GNN maximally exploits the homogeneity and heterogeneity between different variables. Extensive experiments on 8 real-world MTS datasets demonstrate the effectiveness of CrossGNN over existing SOTA methods while maintaining linear memory occupation and computation time as the input size increases. For future work, it is worth exploring the design of dynamic graph networks that can effectively capture complex interactions in out-of-distribution (OOD) scenarios.

References

Rafal A Angryk, Petrus C Martens, Berkay Aydin, Dustin Kempton, Sushant S Mahajan, Sunitha Basodi, Azim Ahmadzadeh, Xumin Cai, Soukaina Filali Boubrahimi, Shah Muhammad Hamdi, et al. Multivariate time series dataset for space weather data analytics. Scientific data , 7(1):227, 2020.

Dawei Cheng, Fangzhou Yang, Sheng Xiang, and Jin Liu. Financial time series forecasting with multi-modality graph neural network. Pattern Recognition , 121:108218, 2022.

Ömer Fahrettin Demirel, Selim Zaim, Ahmet Çali¸ skan, and Pinar Özuyar. Forecasting natural gas consumption in istanbul using neural networks and multivariate time series methods. Turkish Journal of Electrical Engineering and Computer Sciences , 20(5):695-711, 2012.

Zijie Fang, Yang Chen, Yifeng Wang, Zhi Wang, Xiangyang Ji, and Yongbing Zhang. Weaklysupervised semantic segmentation for histopathology images based on dataset synthesis and feature consistency constraint. Proceedings of the AAAI Conference on Artificial Intelligence , 37(1):606-613, Jun. 2023.

Hansika Hewamalage, Christoph Bergmeir, and Kasun Bandara. Recurrent neural networks for time series forecasting: Current status and future directions. International Journal of Forecasting , 37(1):388-427, 2021.

Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In The 41st international ACM SIGIR conference on research & development in information retrieval , pages 95-104, 2018.

Jianxin Li, Qingyun Sun, Hao Peng, Beining Yang, Jia Wu, and S Yu Phillp. Adaptive subgraph neural network with reinforced critical structure mining. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2023.

Zhe Li, Shiyi Qi, Yiduo Li, and Zenglin Xu. Revisiting long-term time series forecasting: An investigation on linear mapping. ArXiv , abs/2305.10721, 2023.

Shengsheng Lin, Weiwei Lin, Wentai Wu, Songbo Wang, and Yongxiang Wang. Petformer: Long-term time series forecasting via placeholder-enhanced transformer, 2023.

Shengsheng Lin, Weiwei Lin, Wentai Wu, Feiyu Zhao, Ruichao Mo, and Haotong Zhang. Segrnn: Segment recurrent neural network for long-term time series forecasting, 2023.

Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li, Weiyao Lin, Alex X Liu, and Schahram Dustdar. Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting. In International conference on learning representations .

Yuqi Nie, Nam H. Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. In International Conference on Learning Representations , 2023.

Andrew Patton. Copula methods for forecasting multivariate time series. Handbook of economic forecasting , 2:899-960, 2013.

P.J. Rousseeuw and A.M. Leroy. Robust regression and outlier detection. Wiley StatsRef: Statistics Reference Online , 1:1-27, 2009.

Amin Shabani, Amir Abdi, Lili Meng, and Tristan Sylvain. Scaleformer: Iterative multi-scale refining transformers for time series forecasting. The Eleventh International Conference on Learning Representations , 2023.

Chao Shang, Jie Chen, and Jinbo Bi. Discrete graph structure learning for forecasting multiple time series. arXiv preprint arXiv:2101.06861 , 2021.

Bowen Shen, Zheng Lin, Yuanxin Liu, Zhengxiao Liu, Lei Wang, and Weiping Wang. COSTEFF: Collaborative optimization of spatial and temporal efficiency with slenderized multi-exit language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pages 1719-1730, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.

Qingyun Sun, Jianxin Li, Hao Peng, Jia Wu, Xingcheng Fu, Cheng Ji, and S Yu Philip. Graph structure learning with variational information bottleneck. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 4165-4174, 2022.

Qingyun Sun, Jianxin Li, Haonan Yuan, Xingcheng Fu, Hao Peng, Cheng Ji, Qian Li, and Philip S Yu. Position-aware structure learning for graph topology-imbalance by relieving under-reaching and over-squashing. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management , pages 1848-1857, 2022.

Tianyu Tang, Jianquan Lu, and Jianlong Qiu. Robust finite-time consensus of coupled markovian logical dynamic networks. IEEE Transactions on Circuits and Systems II: Express Briefs , 2023.

Hao Tian, Xiaolong Xu, Lianyong Qi, Xuyun Zhang, Wanchun Dou, Shui Yu, and Qiang Ni. Copace: Edge computation offloading and caching for self-driving with deep reinforcement learning. IEEE Transactions on Vehicular Technology , 70(12):13281-13293, 2021.

Binwu Wang, Yudong Zhang, Xu Wang, Pengkun Wang, Zhengyang Zhou, Lei Bai, and Yang Wang. Pattern expansion and consolidation on evolving graphs for continual traffic prediction. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 2223-2232, 2023.

Huiqiang Wang, Jian Peng, Feihu Huang, Jince Wang, Junhui Chen, and Yifei Xiao. Micn: Multi-scale local and global context modeling for long-term series forecasting. In The Eleventh International Conference on Learning Representations , 2023.

Xu Wang, Hongbo Zhang, Pengkun Wang, Yudong Zhang, Binwu Wang, Zhengyang Zhou, and Yang Wang. An observed value consistent diffusion model for imputing missing values in multivariate time series. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 2409-2418, 2023.

Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven C. H. Hoi. Etsformer: Exponential smoothing transformers for time-series forecasting. 2022.

Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. Timesnet: Temporal 2d-variation modeling for general time series analysis. In International Conference on Learning Representations , 2023.

Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. Advances in Neural Information Processing Systems , 34:22419-22430, 2021.

Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang, Xiaojun Chang, and Chengqi Zhang. Connecting the dots: Multivariate time series forecasting with graph neural networks. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , 2020.

Xiaolong Xu, Xuyun Zhang, Honghao Gao, Yuan Xue, Lianyong Qi, and Wanchun Dou. Become: Blockchain-enabled computation offloading for iot in mobile edge computing. IEEE Transactions on Industrial Informatics , 16(6):4187-4195, 2020.

Kuo Yang, Zhengyang Zhou, Wei Sun, Pengkun Wang, Xu Wang, and Yang Wang. Extract and refine: Finding a support subgraph set for graph representation. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 2953-2964, 2023.

Yi Yin and Pengjian Shang. Forecasting traffic time series with multivariate predicting method. Applied Mathematics and Computation , 291:266-278, 2016.

Bing Yu, Haoteng Yin, and Zhanxing Zhu. Spatio-temporal graph convolutional networks: A deep learning framework for traffic forecasting. In Proceedings of the 27th International Joint Conference on Artificial Intelligence (IJCAI) , 2018.

Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? arXiv preprint arXiv:2205.13504 , 2022.

Yunhao Zhang and Junchi Yan. Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting. In International Conference on Learning Representations , 2023.

Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. FEDformer: Frequency enhanced decomposed transformer for long-term series forecasting. In Proc. 39th International Conference on Machine Learning (ICML 2022) , 2022.

Zhengyang Zhou, Qihe Huang, Gengyu Lin, Kuo Yang, LEI BAI, and Yang Wang. GReto: Remedying dynamic graph topology-task discordance via target homophily. In The Eleventh International Conference on Learning Representations , 2023.

Zhengyang Zhou, Qihe Huang, Kuo Yang, Kun Wang, Xu Wang, Yudong Zhang, Yuxuan Liang, and Yang Wang. Maintaining the status quo: Capturing invariant relations for ood spatiotemporal learning. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , KDD '23, page 3603-3614, New York, NY, USA, 2023. Association for Computing Machinery.

Zhengyang Zhou, Yang Wang, Xike Xie, Lianliang Chen, and Hengchang Liu. Riskoracle: A minute-level citywide traffic accident forecasting framework. In Proceedings of the AAAI conference on artificial intelligence , volume 34, pages 1258-1265, 2020.

Appendix

A Experimental Details

A.1 Datasets

We conduct extensive experiments on 8 real-world datasets following [27]. The interval length, time step number, and the variable number of each real-world dataset are presented in Table 4. The detailed dataset descriptions are as follows:

1) ETTh (ETTh1, ETTh2, ETTm1, ETTm2) consists of two hourly-level datasets (ETTh) and two 15minute-level datasets (ETTm). Each of them contains seven oil and load features of electricity transformers from July 2016 to July 2018.

2) Weather includes 21 indicators of weather, such as air temperature, and humidity. Its data is recorded every 10 min for 2020 in Germany.

3) Traffic describes hourly road occupancy rates measured by 862 sensors on San Francisco Bay area freeways from 2015 to 2016.

Exchange-rate collects the daily exchange rates of 8 countries from 1990 to 2016.

Electricity contains hourly electricity consumption (in Kwh) of 321 clients from 2012 to 2014.

A.2 Baseline Methods

We briefly describe the selected 9 state-of-the-art baselines as follows:

TimesNet [26] is a task-general foundational model for time series analysis that utilizes a modular architecture to unravel intricate temporal variations. A parameter-efficient inception block is leveraged to capture intra-period and inter-period variations in 2D space.

Crossformer [34] is a versatile vision transformer which solves attention among different variables.

PatchTST [12] is a strong baseline dependent on channel-independence and time series segmentation.

ETSformer [25] is a Transformer-based model for time-series forecasting that incorporates inductive biases of time-series structures and introduces novel exponential smoothing attention (ESA) and frequency attention (FA) to improve performance.

DLinear [33] is a simple linear-based model combined with a decomposition scheme.

FEDformer [35] is a Transformer-based model that uses the seasonal-trend decomposition with frequency-enhanced blocks to capture cross-time dependency for forecasting.

Autoformer [27] is a Transformer-based model using decomposition architecture with an autoCorrelation mechanism to capture cross-time dependency for forecasting.

Pyraformer [11] is a Transformer-based model learning multi-resolution representation of the time series by the pyramidal attention module to capture cross-time dependency for forecasting.

9) MTGNN [28] explicitly utilizes cross-variable dependency using GNN. A graph learning layer learns a graph structure where each node represents one variable in MTS.

A.3 Implementation Details

To ensure a fair comparison, the look-back window size is set to 96, which is consistent with all baselines. We set the scale numbers S to 5 and set K to 10 for all datasets, as sensitivity experiments have shown that S does not have a significant impact beyond 5 and CrossGNN is not sensitive to K . Additionally, the mean squared error (MSE) is used as the loss function. For the learning rate, a grid search is conducted among [5e-3, 1e-3, 5e-4, 1e-4, 5e-5, 1e-5] to obtain the most suitable learning rate for all datasets. Besides, the dimension of the channel is set as 8 for smaller datasets and 16 for larger datasets, respectively. The training would be terminated early if the validation loss does not

decrease for three consecutive rounds. The model is implemented in PyTorch 1.8.0 and trained on a single NVIDIA Tesla V100 PCIe GPU with 16GB memory.

B Additional Experimental Results

B.1 Analysis on Robustness Against Noise

ETTh2

Weather

ETTm2

CrossGNN

CrossGNN

0.6

CrossGNN

0.8

ETSformer

0.45

ETSformer

ETSformer

MTGNN

00

MTGNN

0.5

MTGNN

0.7

0.35

MSE

0.6

MSE

MSE

0.4

0.30

0.5

0.25

0.3

0.4

0.20

0.2

0.3

0.15

101

102

101

102

101

102

db

db

db

To evaluate the robustness of CrossGNN against noise, we add different intensities of Gaussian white noise to the original MTS and observe the performance changes. As the intensity of Gaussian white noise increases, the signal-to-noise ratio (SNR) gradually decreases from 100 dB to 0 dB. Figure 7 shows the MSE results of CrossGNN, ETSformer [25] and MTGNN [28] on ETTh2, ETTTm2, and Weather under different SNR. As the SNR decreases from 100db to 0db, the mean square error (MSE) of CrossGNN increases more slowly than MTGNN and ETSformer.

Taking the results on the ETTm2 dataset as an example, when the noise intensity increases at the beginning (i.e., SNR decreases from 100db to 10db), the prediction accuracy of MTGNN and ETSformer becomes unstable. Their prediction accuracy drops more rapidly when the noise intensity suddenly increases (i.e., SNR decreases from 10db to 0db). In contrast, CrossGNN maintains overall stability, and its performance degrades more slowly. The quantitative results demonstrate that CrossGNN exhibits good robustness against noisy data and has a great advantage when dealing with unexpected fluctuations. Such improvements benefit from the explicit modeling of respective cross-scale and cross-variable interactions.

B.2 Sensitivity Analysis

MSE (Left Y-Axis)

MAE (Right Y-Axis)

ETTh2

ETTm2

Traffic

Weather

0.18

197°0

0.165

0.23

0.58

0.33

0.32

0.36

0.178

0.176

0.262

0.16

0.225

0.578

0.576

0.325

0.3

0.35

0.174

0.26

0.155

0.22

0.574

0.32

0.28

0.34

0.172

0.258

0.572

0.17

0.15

0.215

0.57

0.315

0.26

0.33

0.168

0.256

0.568

0.166

0.254

0.145

0.21

0.566

0.31

0.24

0.32

0.164

0.162

0.252

0.14

0.205

0.564

0.562

0.305

0.22

0.31

0.16

0.25

0.135

0.2

0.56

0.3

0.2

0.3

S

8

5

S

5

S

S

(a)

(b)

(c)

(d)

Number of Scales We vary the number of scales from 4 to 8 and report the MSE and MAE results on ETTh2, ETTm2, Traffic, and Weather. As shown in Figure 8, We observe that the performance improvement becomes less significant after a certain number of scales (i.e., 5), indicating that a certain scale size is sufficient to eliminate most of the effects of temporal noise.

Number of Temporal Node Neighbors The number of temporal neighboring nodes is primarily determined by the hyperparameter K . As depicted in Figure 9, we conducted experiments with

MSE (Left Y-Axis)

MAE (Right Y-Axis)

0.32

ETTh2

0.35

0.18

ETTm2

197°0

Traffic

0.3250.165

Weather

0.23

0.3

0.345

0.34

0.178

0.176

0.262

0.576

0.574

0.32

0.16

0.225

0.28

0.335

0.174

0.26

0.572

0.155

0.22

0.33

0.172

0.258

0.57

0.315

0.26

0.325

0.17

0.15

0.215

0.32

0.168

0.256

0.568

0.31

0.24

0.315

0.166

0.254

0.566

0.145

0.21

0.22

0.305

0.31

0.164

0.162

0.252

0.564

0.562

0.305

0.14

0.205

0.2

0.3

0.16

0.25

0.56

°0

0.135

0.2

10

15

20

25

30

10

15

20

25

30

10

15

20

25

30

10

15

20

25

30

(a)

K

(b)

K

(c)

K

(d)

K

different K values, including 10, 15, 20, 25, and 30, and observed that CrossGNN is not sensitive to the number of K . This suggests that effective cross-scale interaction can be achieved by focusing only on strongly correlated time nodes.

B.3 Visualization of Forecasting Results of Different Models

We present the visualization of forecasting results of CrossGNN and other baseline models on 8 datasets in Figure 10 and Figure 11. These datasets exhibit diverse temporal patterns, with 96-steps input length and output horizon. It can be observed that the prediction results of the Transformerbased model are significantly affected by noise, resulting in fluctuations. In contrast, the prediction results of CrossGNN are less affected by noise, and the predicted values are closer to the true results.

For example, considering the forecasting results on the Traffic dataset, there are three unexpected noise points (i.e., irregularly high points) in the input data. During prediction, the attention mechanism of the Transformer-based model may focus on the noisy points, leading to a bias towards higher output predictions. As a result, although the Transformer-based model seems to capture the periods of the time series, it fails to produce accurate predictions. In contrast, CrossGNN is unaffected by these three noisy data points and generates predictions that are closer to the ground truth. While Transformer-based models struggle to capture the scale and bias of future data due to unexpected noise in the input data, CrossGNN outperforms other models in terms of both scale and bias in forecasting.

C Derivation of Computational Complexity

In this section, we theoretically prove that the time and space complexity of the Cross-Scale module and Cross-Variable module in CrossGNN are both linear. We have organized the notations used in Table 5 for ease of reading.

GroundTruth

Informer

Fedformer

Autoformer

CrossGNN

1.0

1.5

0.8

1.0

0.6

0.5

0.4

0.0

0.2

0.5

0.0

1.0

0.2

1.5

0.4

0

25

50

75.

100.

125

150

175

200

0

25

50

75

100

125

150

175

200

Electricity

Exchange-rate

0.75

4

0.50

3

0.25

0.00

2

0.25

0.50

0

0.75

1.00

1

0

25

50

75

100

125

150

175

200

0

25

50

75

100

125

150

175

200

Traffic

Weather

Proposition 1. The time and space complexity for the Cross-scale GNN is O ( K scale × ln S × L ) and amounts to O ( L ) when S and K scale are constants w.r.t. L .

Proof. To improve readability, we substitute K for K scale . Denote L ( s ) as the number of time nodes at s -th scale:

where p s is the corresponding period length of the s -th scale and L is the original input length (i.e., look-back window size). L ′ is the sum of time nodes at different scales, and it could be expressed by:

where lnS + ϵ + 1 2 S ≈ ∑ S s =1 1 S is the approximate summation formula for the harmonic series, and ϵ is the Euler-Mascheroni constant.

For a time node, we set its scale-sensitive time node neighbor numbers to k s = ⌈ K p s ⌉ at s -th scale. Since the trend-aware neighbor nodes are defined as its previous node and next node at the current scale, the number of trend-aware neighbor nodes can reach 2 when these nodes do not overlap with the scale-sensitive neighbor nodes. However, when there is overlap, the number of trend-aware neighbor nodes can be 0 or 1.Therefore, the maximum neighbor node number of v i is given by:

GroundTruth

Informer

Fedformer

Autoformer

CrossGNN

1.5

1.5

1.0

1.0

0.5

0.0

0.5

0.5

0.0

1.0

1.5

0.5

2.0

0

25

50

75

100

125

150

175

200

0

25

50

75

100

125

150

175

200

ETThl

ETTh2

1.0

几

0.5

0

hmuw

0.0

0.5

1

-1.0

1.5

2.0

-3

0

25

50

75

100

125

150

175

200

0

25

50

75

100

125

150

175

200

ETTml

ETTm2

Total correlated node pair number is expressed as:

Consequently, the complexity of the proposed cross-scale GNN is:

Since K and S are all constant terms that are independent of the length L and remain fixed when L changes, the complexity can be further reduced to O ( L ) .

Proposition 2. The time and space complexity for the Cross-variable GNN is O ( K var × D ) and amounts to O ( D ) when K var is a constant w.r.t. D .

Proof. Without loss of generality, we assume that the number of homogeneous and heterogeneous correlated nodes for each variable are both K var . For a cross-variable graph, there are a total of

∑ D i =1 2 K var = 2 K var D correlated variable node pairs. Correspondingly, since the complexity of graph computation is related to the number of edges, the time and space complexity of cross-variable GNNare both O ( K var × D ) . As K var is a constant that is independent of D , its complexity is linear (i.e., O ( D ) ).

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work first analyzes real-world datasets for multivariate time series forecasting and finds out two problems that are not well handled by previous works: 1) unexpected noise; 2) heterogeneity between variables. A GNN-based model, named CrossGNN, is proposed to fill the gap. CrossGNN consists of three components: 1)Adaptive Multi-Scale Identifier identifies potential periods with FFT and aggregates time series at different scales to construct a mult-scale view of the data; 2)Cross-Scale GNN constructs a graph among scales and uses GNN to capture cross-scale dependency; 3)Cross-Variable GNN captures homogeneous and heterogeneous dependency among variables. Correlation graphs used in Cross-Scale and Cross-Variable GNNs are restricted to be sparse, so the complexity is linear to the input length. Experiments on 8 real-world datasets demonstrate the effectiveness of the proposed model.\n\nSTRENGTHS:\n- This work is well-structured and easy to follow.\n- The studied problem, noise in datasets, is an important and practical problem for MTS forecasting.\n- Three components in CrossGNN are well-motivated and reasonable. The linear complexity is also an advantage.\n- Robustness analysis of noise in section 4.3 is interesting.\n\nWEAKNESSES:\n- My main concern is that graphs used in Cross-Scale and Cross-Variable GNNs are static for each dataset. These graphs are constructed by shared learnable vectors with filtering, so they are static and the same for different inputs. This leads to 1) the dependency structure of time steps being fully determined by the positions in the series; 2) the dependency structure of variables being the same for different inputs in a dataset. \n- Some recent works (e.g. PatchTST[1], Crossformer[2]) that explicitly model the cross-variable dependency should be compared in the main experiment.\n\n[1]Yuqi Nie, Nam H. Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. In International Conference on Learning Representations, 2023.\n\n[2]Yunhao Zhang and Junchi Yan. Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting. In International Conference on Learning Representations, 2023.\n\nQUESTIONS:\n- In Equation (1), amplitudes are averaged over variables. Is the average operation reasonable? What if different variables have different periodic patterns?\n- How is the ouput of AMSI (with shape $L' \\times D$) embedded into latent vectors with shape $L' \\times D \\times C$?\n- Settings of the ablation study should be described in detail, perhaps in the appendix. What does \"divides the scales directly by fixed length\" mean? Does C-Hete only remove the heterogeneous connections? If so, an ablation study that removes both homogeneous and heterogeneous connections should be conducted.\n- It is better to compare the efficiency with baselines such as DLinear, Pyraformer, and TimesNet in Figure 6, as it is obvious that the proposed model is more efficient than the full graph. Moreover, the complexity of Pyraformer is $O(L)$, not $O(L \\log L)$.\n- See my other questions in the weakness part.\n\nLIMITATIONS:\nNo potential negative societal impacts.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-24"}

comments: SUMMARY:
CrossGNN is a linear complexity GNN model designed for MTS forecasting, addressing two obstacles: self-attention mechanisms assigning high scores to outlier points and real-world data homogeneity and heterogeneity. By combining Adaptive Multi-Scale Identifier (AMSI), Cross-Scale GNN, and Cross-variable GNN, CrossGNN outperforms recent SOTA methods in real-world datasets.

STRENGTHS:
1. Despite considering the relationship between variables, unlike Transformer-based models, it has a low time complexity O(L).
2. The paper also includes a comprehensive set of ablation studies and the performance in forecasting is SOTA.

WEAKNESSES:
1. The authors mentioned that noise can disturb attention mechanism in transformer. However, the noise ratio is less than 2\% as shown in Figure 1 (b) and I do not think this would be significant. Also, those outliers can be removed by using outlier detection techniques before training. 
2. It is necessary to add an explanation or experiments to assure that the self-attention mechanism really adversely affects performance by assigning outlier points as high scores.
3. Technical novelty is a bit weak. Concepts of cross-scale and cross-variable are already introduced concepts in TimesNet and Crossformer. Also, many papers have already used GNN to deal with cross-variable interaction. What is the main difference except the input?

QUESTIONS:
In Figure 2, all scales are concatenated in time dimension from coarse to fine scale. Why this kind of concatenation is needed? Is there any reason for this?

LIMITATIONS:
There seems no limitation in this paper. Please include one.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nOverall Comment：\nThis article addresses two issues in multi-variate time-series modeling: i) How to address signal noise in multivariate time series, and ii) How to address interactions between multiple variables to extract information. The article proposes two GNN models to solve these problems, including the Cross-Scale GNN model for addressing signal noise and the Cross-Variable GNN for addressing interactions between different signals.\n\nOverall, while some parts of the article are not very clear, such as, what is the meaning of  homogeneity and heterogeneity in time series, and the the ablation study can be improved, the article's innovative ideas are clear and worth being seen by more people. Compared to conventional baseline methods, this article has significant advantages.\n\nThe strengths of the article include:\n\n1. The paper conducts a large number of experiments, tested on 8 datasets. From the experimental results, the proposed method is effective and performs better than conventional MTS baseline methods.\n2. The article is well written and easy to read, making it easy to understand the information that the author is trying to convey.\n\nThe weaknesses of the article include:\n\n1. The analysis of the ablation study needs to clarify that which module, the Cross-Scale GNN and Cross-Variable GNN, plays a larger role in the model in order to judge whether it is filtering signal noise that makes the proposed model perform better on each dataset or whether introducing interactions between multiple variables is the key factor.\n2. Correspondingly, if the Cross-Scale GNN model for addressing signal noise plays a larger role, is the focus of the article on time series denoising or modeling time series with noise? The article should be compared with methods specifically designed for denoising time series, rather than just conventional time series modeling baselines.\n3. What is the meaning of homogeneity and heterogeneity in time series signals? In Figure 1(d), only positive or negative correlations between variables are displayed, which is not related to homogeneity and heterogeneity relationships between variables.\n\nSTRENGTHS:\nThe strengths of the article include:\n\n1. The paper conducts a large number of experiments, tested on 8 datasets. From the experimental results, the proposed method is effective and performs better than conventional MTS baseline methods.\n2. The article is well written and easy to read, making it easy to understand the information that the author is trying to convey.\n\nWEAKNESSES:\nThe weaknesses of the article include:\n\n1. The analysis of the ablation study needs to clarify that which module, the Cross-Scale GNN and Cross-Variable GNN, plays a larger role in the model in order to judge whether it is filtering signal noise that makes the proposed model perform better on each dataset or whether introducing interactions between multiple variables is the key factor.\n2. Correspondingly, if the Cross-Scale GNN model for addressing signal noise plays a larger role, is the focus of the article on time series denoising or modeling time series with noise? The article should be compared with methods specifically designed for denoising time series, rather than just conventional time series modeling baselines.\n3. What is the meaning of homogeneity and heterogeneity in time series signals? In Figure 1(d), only positive or negative correlations between variables are displayed, which is not related to homogeneity and heterogeneity relationships between variables.\n\nQUESTIONS:\nAs I write in the Weaknesses: \n\n1. the Cross-Scale GNN and Cross-Variable GNN, which plays a larger role in the model？ Whether it is filtering signal noise that makes the proposed model perform better on each dataset or whether introducing interactions between multiple variables is the key factor?\n\n2. What is the meaning of homogeneity and heterogeneity in time series signals? In Figure 1(d), only positive or negative correlations between variables are displayed, which is not related to homogeneity and heterogeneity relationships between variables.\n\nLIMITATIONS:\nThe authors didn't address the limitations clearly.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

comments: SUMMARY:
This paper aims to deal with the temporal fluctuations and heterogeneity between variables, caused by unexpected noise, for better multivariate time-series forecasting. Specifically, the authors propose a linear complexity CrossGNN model, including Cross-Scale GNN which captures relationships inter- and intra- scales, and Cross-Variable GNN which captures the homogeneity and heterogeneity relationships between different variables. Experiments on 8 benchmark multivariate time-series datasets demonstrate the effectiveness of CrossGNN over some existing methods.

STRENGTHS:
1. The authors propose a GNN-based (CrossGNN) method with linear complexity for long-term time series forecasting.
2. The CrossGNN captures the relationships between both scales and variables.
3. The paper is easy to understand.

WEAKNESSES:
1. The proposed CrossGNN does not compare with the SoTA methods, like RLinear, RMLP [1] and PatchTST [2]. I checked the results in this paper (Table 1 & Figure 4) and [1,2], and found that CrossGNN is worse than RLinear, RMLP and PatchTST.

[1] Revisiting Long-term Time Series Forecasting: An Investigation on Linear Mapping

[2] A Time Series is Worth 64 Words: Long-term Forecasting with Transformers

2. It is unclear why the heterogeneity between variables is caused by unexpected noise.
3. In the top of Page 5, it is unclear why the production of two learnable vectors can diminish the effect of noise.
4. What is the meaning of "C" in Figure 2?

QUESTIONS:
1. Please compare with SoTA methods, like RLinear, RMLP and PatchTST.
2. Please give explanations on the weaknesses 2~4.

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-06