SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Cross-variable Interaction. For variable  $v_{i}$ , the disentangled cross-variable message passing can be formulated as:
$$
\mathcal {H N} _ {:, i} ^ {v a r, N} = \sum_ {v _ {j} \in \mathcal {N} _ {+} ^ {v a r} (v _ {i})} E ^ {v a r} [ i, j ] \cdot \mathcal {H} _ {:, j} ^ {v a r, N - 1} + \sum_ {v _ {k} \in \mathcal {N} _ {-} ^ {v a r} (v _ {i})} E ^ {v a r} [ i, k ] \cdot \mathcal {H} _ {:, k} ^ {v a r, N - 1},
$$
$$
\mathcal {H} _ {:, i} ^ {\text {v a r}, N} = \sigma \left(\operatorname {C o n c a t} \left(\mathcal {H N} _ {:, i} ^ {\text {v a r}, N}, \mathcal {H} _ {:, i} ^ {\text {v a r}, N - 1}\right) \cdot W\right), \tag {13}
$$
$$
\mathcal {H} _ {:, i} ^ {v a r, N} = \mathcal {H} _ {:, i} ^ {v a r, N} / | | \mathcal {H} _ {:, i} ^ {v a r, N} | | _ {2},
$$
$\mathcal{H}^{var}$  is the variable node feature;  $\mathcal{HN}^{var}$  is the aggregation of neighbor variable node feature.  $\mathcal{HN}_{:,i}^{var,N}$  aggregates both homogeneous and heterogeneous neighbor node feature of previous layer correlated to  $v_{i}^{var}$ .  $\mathcal{H}_{:,i}^{var,N}$  is updated by the aggregated variable feature  $\mathcal{HN}_{:,i}^{var,N}$  and its previous layer feature  $\mathcal{H}_{:,i}^{var,N-1}$ . Finally, the normalized  $\mathcal{H}_{:,i}^{var,N}$  at  $N$ -th layer is the output of Cross-variable GNN.
3.4 Direct Multi-step forecasting
After obtaining the output features of Cross-Variable GNN, we exploit the direct multi-step (DMS) forecasting [33] for the decoder to predict the multi-step MTS at once. We take two MLPs as the decoder, where the first  $\mathrm{MLP}_{\mathrm{C}}$  maps the time dimension of features from  $C$  to 1, while the second  $\mathrm{MLP}_{\mathrm{T}}$  maps the time dimension from the historical input sequence  $L'$  to the output sequence length. The final prediction can be obtained by:
$$
\left\{\hat {X} _ {1} ^ {t}, \dots , \hat {X} _ {D} ^ {t} \right\} _ {t = L + 1} ^ {L + T} = \mathrm {M L P} _ {\mathrm {T}} \left(\mathrm {M L P} _ {\mathrm {C}} \left(\mathcal {H} ^ {v a r}\right)\right). \tag {14}
$$
4.1 Datasets And Experiment Setup
Datasets We conduct extensive experiments on 8 real-world datasets following [27], including Weather, Traffic, Exchange Rate, Electricity and 4 ETT datasets(ETTh1, ETTh2, ETTm1 and ETTm2). We follow the standard protocol in [27] and split datasets into training, validation and test set by the ratio of 6:2:2 for the last 4 ETT datasets, and 7:1:2 for the other datasets.
Baselines and Setup We compare our method with 9 state-of-the-art methods, including TimesNet [26]; 6 Transformer-based methods: PatchTST [12], Crossformer [34], ETSformer [25], FEDformer [35], Pyraformer [11], Autoformer [27]; GNN-based method: MTGNN [28]; simple yet powerful linear model Dlinear [33]. All the models are following the same experimental setup with prediction length  $T \in \{96, 192, 336, 720\}$  for all datasets as in the original papers. We collect all baseline results from [26] except MTGNN, with the default look-back window  $L = 96$ . We
Table 2: Performance comparisons on ablative variants
Datasets	Weather	Traffic	ETTm2
Predict Length	96	192	336	720	96	192	336	720	96	192	336	720
C-AMSI	MSE	0.167	0.220	0.276	0.358	0.585	0.590	0.596	0.610	0.192	0.249	0.313	0.420
MAE	0.229	0.284	0.318	0.382	0.330	0.336	0.341	0.354	0.279	0.318	0.358	0.417
C-CS	MSE	0.175	0.231	0.290	0.371	0.588	0.599	0.603	0.614	0.200	0.259	0.328	0.422
MAE	0.241	0.286	0.325	0.385	0.331	0.343	0.342	0.358	0.281	0.325	0.364	0.421
C-Hete	MSE	0.172	0.224	0.277	0.364	0.589	0.588	0.603	0.616	0.190	0.255	0.320	0.417
MAE	0.231	0.280	0.327	0.381	0.329	0.332	0.335	0.354	0.280	0.320	0.358	0.417
C-CV	MSE	0.174	0.227	0.279	0.367	0.588	0.591	0.604	0.616	0.193	0.257	0.322	0.419
MAE	0.235	0.281	0.328	0.382	0.330	0.335	0.338	0.358	0.278	0.323	0.359	0.418
CrossGNN	MSE	0.159	0.211	0.267	0.352	0.570	0.577	0.588	0.597	0.176	0.240	0.304	0.406
MAE	0.218	0.266	0.307	0.362	0.310	0.321	0.324	0.337	0.265	0.307	0.345	0.400
reproduced the results of MTGNN [28] with look-back window  $L = 96$  on all datasets according to the settings in the original paper. We calculate the Mean Square Error(MSE) and Mean Absolute Error(MAE) of MTS forecasting as metrics. More details about datasets, baselines, implementation, hyper-parameters are shown in Appendix A.3.
The quantitative results of MTS forecasting using different methods is shown in Table 1. CrossGNN achieves outstanding performance on most datasets across various prediction length settings, obtaining 47 first-place and 9 second-place rankings in total 64 settings. Quantitatively, compared with the best results that Transformer-based methods can offer, CrossGNN achieves an overall  $10.43\%$  reduction on MSE and  $10.11\%$  reduction on MAE. Compared with GNN-based method MTGNN, CrossGNN achieves a more significant reduction  $22.57\%$  on MSE and  $25.74\%$  on MAE. Compared with other strong baselines like TimesNet and Dlinear, CrossGNN can still outperform them in general. Our method dose not achieve the best performance on the Electricity dataset. Further analysis reveals that the more severe out-of-distribution (OOD) problem in Electricity dataset results in a lower generalization of the learned temporal graph relations on the test set.
4.3 Robustness Analysis of Noise
To evaluate the model robustness against noise, we add different intensities of Gaussian white noise to the original MTS and observe the performance changes of different methods.
Figure 3 shows the MSE results of CrossGNN, ETSformer [25] and MTGNN [28] under different noise ratio on ETTm2 dataset, and the input length is set as 96. As the signal-to-noise ratio (SNR) decreases from 100db to 0db, the mean square error (MSE) increases more slowly on CrossGNN (0.177) compared to ETSformer (0.191) and MTGNN (0.205). The quantitative results demonstrate that CrossGNN exhibits good robustness against noisy data and has a great advantage when dealing with unexpected fluctuations. We speculate such improvements benefit from the explicit modeling of respective scale-level and variable-level interactions.
Figure 3: Robustness analysis under different signal-to-noise ratio (SNR) on ETTm2.
We conduct ablation studies by removing corresponding modules from CrossGNN on three datasets. C-AMSI removes the adaptive multi-scale identifier (AMSI) and directly use  $k$  fixed lengths (e.g., 1, 2, 3, ...,  $k$ ) for average pooling. C-CS removes the Cross-Scale GNN module. C-Hete removes the heterogeneous connections and focuses on the homogeneous correlation modeling between different variables. C-CV removes both homogeneous and heterogeneous connections. We analyze the results shown in Table 2. Obs.1) Removing
Cross-Scale GNN results in the most significant decrease in prediction metrics, emphasizing its strong ability in modeling the interaction between different scales and time points. Obs.2) Cross-Variable GNN also improves the model performance a lot, demonstrating the importance of modeling the complex and dynamic interaction between different variables. Obs.3) AMSI constantly improves the forecasting accuracy, suggesting that different scales of MTS contain rich interaction information.
4.5 Hyper-Parameter sensitivity
Look-Back Window Size Figure 4 shows the MSE results of models with different look-back window sizes on four datasets. As the window size increases, the performance of Transformer-based models fluctuates while CrossGNN constantly improves. This indicates that the attention mechanism of Transformer-based methods may focus much more on the temporal noise but our method can better extract the relationships between different time nodes through Cross-Scale GNN.
Number of Scales We vary the number of scales from 4 to 8 and report the MSE and MAE results on Weather and Traffic dataset. As shown in Figure 5 (a) and Figure 5 (b), We observe that the performance improvement becomes less significant after a certain number of scales, indicating that a certain scale size is sufficient to eliminate most of the effects of temporal noise. Number of Node Neighbors The number of neighboring nodes limited to each time node is mainly determined by the hyperparameter  $K$ . As shown on Figure 5 (c) and Figure 5 (d), we experiment with  $K$  values of 10, 15, 20, 25, and 30 and found that CrossGNN is not sensitive to the number of  $K$ . This indicates that it is only necessary to focus on strongly correlated nodes for effective information aggregation in temporal interaction.
Figure 4: The MSE results (Y-axis) of models with different look-back window sizes (X-axis) on ETTh2, ETTm2, Traffic and Weather, the output length is set as 336.
Figure 5: The MSE (left Y-axis) and MAE results (right Y-axis) of CrossGNN on Traffic and Weather. (a) and (b) display the performance on different scale numbers. (c) and (d) demonstrates the performance on different number of node neighbors
4.6 Complexity Analysis
Table 3 illustrates the theoretical complexity of CrossGNN and existing Transformer-based methods. Detailed complexity derivation can be found in the Appendix C. To verify that the time and space complexity of our method is indeed  $O(L)$ , we use TVM to implement the GNN computation part and compare the computation time and memory usage of with full connected graph during inference on ETTh2. Comparison experiments are implemented on a Intel(R) 8255C CPU @ 2.50GHZ with 40GB memory, centos 7.8, and TVM 1.0.0. Figure 6 illustrates the time and memory cost of GNN modules, and our proposed approach is close to linear with the input length.
Method	Complexity per layer
Transformer	O(L2)
Informer	O(L log L)
Autoformer	O(L log L)
Pyraformer	O(L)
CrossGNN	O(L)
Table 3: The computation complexity in theory.  $L$  represents the length of the input data.
Figure 6: Comparison of time and memory consumption on ETTh2. (a) memory occupation. (b) computation time.
In this work, we construct a comprehensive analysis of real-world data and observe that the temporal fluctuations and heterogeneity between variables, caused by unexpected noise, are not well handled by current popular time-series forecasting methods. To address above issues, we propose a linear complexity CrossGNN model, which is the first GNN model to refine both cross-scale and cross-variable interaction for MTS forecasting. An adaptive multi-scale identifier (AMSI) is leveraged to obtain multi-scale time series with reduced noise from input MTS. In particular, Cross-Scale GNN captures the scales with clearer trend and weaker noise, while Cross-Variable GNN maximally exploits the homogeneity and heterogeneity between different variables. Extensive experiments on 8 real-world MTS datasets demonstrate the effectiveness of CrossGNN over existing SOTA methods while maintaining linear memory occupation and computation time as the input size increases. For future work, it is worth exploring the design of dynamic graph networks that can effectively capture complex interactions in out-of-distribution (OOD) scenarios.
This paper is partially supported by the National Natural Science Foundation of China (No.62072427, No.12227901), the Project of Stable Support for Youth Team in Basic Research Field, CAS (No.YSBR-005), Academic Leaders Cultivation Program, USTC.
[1] Rafal A Angryk, Petrus C Martens, Berkay Aydin, Dustin Kempton, Sushant S Mahajan, Sunitha Basodi, Azim Ahmadzadeh, Xumin Cai, Soukaina Filali Boubrahimi, Shah Muhammad Hamdi, et al. Multivariate time series dataset for space weather data analytics. Scientific data, 7(1):227, 2020.
[2] Dawei Cheng, Fangzhou Yang, Sheng Xiang, and Jin Liu. Financial time series forecasting with multi-modality graph neural network. Pattern Recognition, 121:108218, 2022.
[3] Ömer Fahrettin Demirel, Selim Zaim, Ahmet Çalıkan, and Pinar Özuyar. Forecasting natural gas consumption in istanbul using neural networks and multivariate time series methods. Turkish Journal of Electrical Engineering and Computer Sciences, 20(5):695-711, 2012.
[4] Zijie Fang, Yang Chen, Yifeng Wang, Zhi Wang, Xiangyang Ji, and Yongbing Zhang. Weakly-supervised semantic segmentation for histopathology images based on dataset synthesis and feature consistency constraint. Proceedings of the AAAI Conference on Artificial Intelligence, 37(1):606-613, Jun. 2023.
[5] Hansika Hewamalage, Christoph Bergmeir, and Kasun Bandara. Recurrent neural networks for time series forecasting: Current status and future directions. International Journal of Forecasting, 37(1):388-427, 2021.
[6] Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In The 41st international ACM SIGIR conference on research & development in information retrieval, pages 95-104, 2018.
We conduct extensive experiments on 8 real-world datasets following [27]. The interval length, time step number, and the variable number of each real-world dataset are presented in Table 4. The detailed dataset descriptions are as follows:
1) ETTh (ETTh1, ETTh2, ETTm1, ETTm2) consists of two hourly-level datasets (ETTh) and two 15minute-level datasets (ETTm). Each of them contains seven oil and load features of electricity transformers from July 2016 to July 2018.
2) Weather includes 21 indicators of weather, such as air temperature, and humidity. Its data is recorded every  $10\mathrm{min}$  for 2020 in Germany.
3) Traffic describes hourly road occupancy rates measured by 862 sensors on San Francisco Bay area freeways from 2015 to 2016.
4) Exchange-rate collects the daily exchange rates of 8 countries from 1990 to 2016.
5) Electricity contains hourly electricity consumption (in Kwh) of 321 clients from 2012 to 2014.
Table 4: The statistics of the datasets for MTS forecasting.
Datasets	ETTh1	ETTh2	ETTm1	ETTm2	Weather	Traffic	Exchange-rate	Electricity
Interval Length	1 Hour	1 Hour	15 Minutes	15 Minutes	10 Minutes	1 Hour	1 Day	1 Hour
Time step #	17,420	17,420	69,680	69,680	52,696	17,544	7,588	26,304
Variable #	7	7	7	7	21	862	8	321
A.2 Baseline Methods
We briefly describe the selected 9 state-of-the-art baselines as follows:
1) TimesNet [26] is a task-general foundational model for time series analysis that utilizes a modular architecture to unravel intricate temporal variations. A parameter-efficient inception block is leveraged to capture intra-period and inter-period variations in 2D space.
2) Crossformer [34] is a versatile vision transformer which solves attention among different variables.
3) PatchTST [12] is a strong baseline dependent on channel-independence and time series segmentation.
4) ETSformer [25] is a Transformer-based model for time-series forecasting that incorporates inductive biases of time-series structures and introduces novel exponential smoothing attention (ESA) and frequency attention (FA) to improve performance.
5) DLinear [33] is a simple linear-based model combined with a decomposition scheme.
6) FEDformer [35] is a Transformer-based model that uses the seasonal-trend decomposition with frequency-enhanced blocks to capture cross-time dependency for forecasting.
7) Autoformer [27] is a Transformer-based model using decomposition architecture with an autoCorrelation mechanism to capture cross-time dependency for forecasting.
8) Pyraformer [11] is a Transformer-based model learning multi-resolution representation of the time series by the pyramidal attention module to capture cross-time dependency for forecasting.
9) MTGNN [28] explicitly utilizes cross-variable dependency using GNN. A graph learning layer learns a graph structure where each node represents one variable in MTS.
A.3 Implementation Details
To ensure a fair comparison, the look-back window size is set to 96, which is consistent with all baselines. We set the scale numbers  $S$  to 5 and set  $K$  to 10 for all datasets, as sensitivity experiments have shown that  $S$  does not have a significant impact beyond 5 and CrossGNN is not sensitive to  $K$ . Additionally, the mean squared error (MSE) is used as the loss function. For the learning rate, a grid search is conducted among [5e-3, 1e-3, 5e-4, 1e-4, 5e-5, 1e-5] to obtain the most suitable learning rate for all datasets. Besides, the dimension of the channel is set as 8 for smaller datasets and 16 for larger datasets, respectively. The training would be terminated early if the validation loss does not
decrease for three consecutive rounds. The model is implemented in PyTorch 1.8.0 and trained on a single NVIDIA Tesla V100 PCIe GPU with 16GB memory.
B.1 Analysis on Robustness Against Noise
Figure 7: The MSE results (Y-axis) of models on ETTh2, ETTm2 and Weather with different signal-to-noise ratio (SNR).
To evaluate the robustness of CrossGNN against noise, we add different intensities of Gaussian white noise to the original MTS and observe the performance changes. As the intensity of Gaussian white noise increases, the signal-to-noise ratio (SNR) gradually decreases from 100 dB to 0 dB. Figure 7 shows the MSE results of CrossGNN, ETSformer [25] and MTGNN [28] on ETTh2, ETTm2, and Weather under different SNR. As the SNR decreases from 100db to 0db, the mean square error (MSE) of CrossGNN increases more slowly than MTGNN and ETSformer.
Taking the results on the ETTm2 dataset as an example, when the noise intensity increases at the beginning (i.e., SNR decreases from 100db to 10db), the prediction accuracy of MTGNN and ETSformer becomes unstable. Their prediction accuracy drops more rapidly when the noise intensity suddenly increases (i.e., SNR decreases from 10db to 0db). In contrast, CrossGNN maintains overall stability, and its performance degrades more slowly. The quantitative results demonstrate that CrossGNN exhibits good robustness against noisy data and has a great advantage when dealing with unexpected fluctuations. Such improvements benefit from the explicit modeling of respective cross-scale and cross-variable interactions.
B.2 Sensitivity Analysis
Figure 8: The MSE (left Y-axis) and MAE results (right Y-axis) of CrossGNN with different number of scales (X-axis) on ETTh2, ETTm2, Traffic, and Weather.
Number of Scales We vary the number of scales from 4 to 8 and report the MSE and MAE results on ETTh2, ETTm2, Traffic, and Weather. As shown in Figure 8, We observe that the performance improvement becomes less significant after a certain number of scales (i.e., 5), indicating that a certain scale size is sufficient to eliminate most of the effects of temporal noise.
Number of Temporal Node Neighbors The number of temporal neighboring nodes is primarily determined by the hyperparameter  $K$ . As depicted in Figure 9, we conducted experiments with
Figure 9: The MSE (left Y-axis) and MAE results (right Y-axis) of CrossGNN with different K (X-axis) on ETTh2, ETTm2, Traffic, and Weather.
different  $K$  values, including 10, 15, 20, 25, and 30, and observed that CrossGNN is not sensitive to the number of  $K$ . This suggests that effective cross-scale interaction can be achieved by focusing only on strongly correlated time nodes.
B.3 Visualization of Forecasting Results of Different Models
We present the visualization of forecasting results of CrossGNN and other baseline models on 8 datasets in Figure 10 and Figure 11. These datasets exhibit diverse temporal patterns, with 96-steps input length and output horizon. It can be observed that the prediction results of the Transformer-based model are significantly affected by noise, resulting in fluctuations. In contrast, the prediction results of CrossGNN are less affected by noise, and the predicted values are closer to the true results.
For example, considering the forecasting results on the Traffic dataset, there are three unexpected noise points (i.e., irregularly high points) in the input data. During prediction, the attention mechanism of the Transformer-based model may focus on the noisy points, leading to a bias towards higher output predictions. As a result, although the Transformer-based model seems to capture the periods of the time series, it fails to produce accurate predictions. In contrast, CrossGNN is unaffected by these three noisy data points and generates predictions that are closer to the ground truth. While Transformer-based models struggle to capture the scale and bias of future data due to unexpected noise in the input data, CrossGNN outperforms other models in terms of both scale and bias in forecasting.
C Derivation of Computational Complexity
In this section, we theoretically prove that the time and space complexity of the Cross-Scale module and Cross-Variable module in CrossGNN are both linear. We have organized the notations used in Table 5 for ease of reading.
Table 5: Meaning of notations
Notation	Meaning
vi	The i-th time node
S	Number of scales
s	Index of the scale
ps	Period length of the s-th scale
L	Original input length (i.e., look-back window size)
L(s)	Time length at the s-th scale
L'	Total length of concatenated multi-scale time series
Kscale	The hyperparameter to control temporal neighbor numbers
Kvar	The hyperparameter to control variable neighbor numbers
ks	The temporal neighbor numbers at the s-th scale
A(vi)	Total temporal neighbor node number correlated to vi
A	Total correlated temporal node pair number
D	Variable numbers
Figure 10: Visualization of 96-step forecasting results on Electricity, Exchange-rate, Traffic, and Weather, and the look-back window size is set as 96.
Proposition 1. The time and space complexity for the Cross-scale GNN is  $O(K^{scale} \times \ln S \times L)$  and amounts to  $O(L)$  when  $S$  and  $K^{scale}$  are constants w.r.t.  $L$ .
Proof. To improve readability, we substitute  $K$  for  $K^{scale}$ . Denote  $L(s)$  as the number of time nodes at  $s$ -th scale:
$$
L (s) = \left\lfloor \frac {L}{p _ {s}} \right\rfloor , 1 \leq s \leq S, \tag {15}
$$
where  $p_{s}$  is the corresponding period length of the  $s$ -th scale and  $L$  is the original input length (i.e., look-back window size).  $L'$  is the sum of time nodes at different scales, and it could be expressed by:
$$
L ^ {\prime} = \sum_ {s = 1} ^ {S} L (s) = \sum_ {s = 1} ^ {S} \left\lfloor \frac {L}{p _ {s}} \right\rfloor \leq \sum_ {s = 1} ^ {S} \left\lfloor \frac {L}{s} \right\rfloor \leq L \sum_ {s = 1} ^ {S} \frac {1}{s} \approx L (\ln S + \epsilon + \frac {1}{2 S}), \tag {16}
$$
where  $\ln S + \epsilon + \frac{1}{2S} \approx \sum_{s=1}^{S} \frac{1}{S}$  is the approximate summation formula for the harmonic series, and  $\epsilon$  is the Euler-Mascheroni constant.
For a time node, we set its scale-sensitive time node neighbor numbers to  $k_{s} = \lceil \frac{K}{p_{s}} \rceil$  at  $s$ -th scale. Since the trend-aware neighbor nodes are defined as its previous node and next node at the current scale, the number of trend-aware neighbor nodes can reach 2 when these nodes do not overlap with the scale-sensitive neighbor nodes. However, when there is overlap, the number of trend-aware neighbor nodes can be 0 or 1. Therefore, the maximum neighbor node number of  $v_{i}$  is given by:
Figure 11: Visualization of 96-step forecasting results on ETTh1, ETTh2, ETTm1 and ETTm2, and the look-back window size is set as 96.
$$
\begin{array}{l} A \left(v _ {i}\right) \leq \sum_ {s = 1} ^ {S} k _ {s} + 2 (17) \\ = K \sum_ {s = 1} ^ {s = S} \frac {1}{p (s)} + 2 (18) \\ \leq K \left(\frac {1}{1} + \frac {1}{2} + \dots + \frac {1}{S}\right) + 2 (19) \\ \approx K (\ln S + \epsilon + \frac {1}{2 S}) + 2, (20) \\ \end{array}
$$
Total correlated node pair number is expressed as:
$$
A = L ^ {\prime} \times A \left(v _ {i}\right) \leq L \left(\ln S + \epsilon + \frac {1}{2 S}\right) \left(K \left(\ln S + \epsilon + \frac {1}{2 S}\right) + 2\right) \approx 2 K \times \ln (S) \times L. \tag {21}
$$
Consequently, the complexity of the proposed cross-scale GNN is:
$$
O (A) \leq O (2 K \times \ln (S) \times L). \tag {22}
$$
Since  $K$  and  $S$  are all constant terms that are independent of the length  $L$  and remain fixed when  $L$  changes, the complexity can be further reduced to  $O(L)$ .
Proposition 2. The time and space complexity for the Cross-variable GNN is  $O(K^{var} \times D)$  and amounts to  $O(D)$  when  $K^{var}$  is a constant w.r.t.  $D$ .
Proof. Without loss of generality, we assume that the number of homogeneous and heterogeneous correlated nodes for each variable are both  $K^{var}$ . For a cross-variable graph, there are a total of