SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Qihe Huang†,§, Lei Shen§, Ruixin Zhang§, Shouhong Ding§, Binwu Wang†, Zhengyang Zhou†,‡, Yang Wang†,‡,
† University of Science and Technology of China (USTC), Hefei, China ‡ Suzhou Institute for Advanced Research, USTC, Suzhou, China § Youtu Laboratory, Tencent, Shanghai, China Email: {hqh, wbw1995} @ mail.ustc.edu.cn, {zzy0929, angyan} @ ustc.edu.cn, shenlei1996@gmail.com, {ruixinzhang, ericshding} @ Tencent.com
Recently, multivariate time series (MTS) forecasting techniques have seen rapid development and widespread applications across various fields. Transformer-based and GNN-based methods have shown promising potential due to their strong ability to model interaction of time and variables. However, by conducting a comprehensive analysis of the real-world data, we observe that the temporal fluctuations and heterogeneity between variables are not well handled by existing methods. To address the above issues, we propose CrossGNN, a linear complexity GNN model to refine the cross-scale and cross-variable interaction for MTS. To deal with the unexpected noise in time dimension, an adaptive multi-scale identifier (AMSI) is leveraged to construct multi-scale time series with reduced noise. A Cross-Scale GNN is proposed to extract the scales with clearer trend and weaker noise. Cross-Variable GNN is proposed to utilize the homogeneity and heterogeneity between different variables. By simultaneously focusing on edges with higher saliency scores and constraining those edges with lower scores, the time and space complexity (i.e.,  $O(L)$ ) of CrossGNN can be linear with the input sequence length  $L$ . Extensive experimental results on 8 real-world MTS datasets demonstrate the effectiveness of CrossGNN compared with state-of-the-art methods. The code is available at https://github.com/hqh0728/CrossGNN.
Time series forecasting has been widely used in many fields (i.e., climate [1], traffic [31], energy [3], finance [13], etc) [8, 29, 20, 10]. Multivariate time series (MTS) consists of time series with multiple variables and MTS forecasting aims at predicting future values based on historical time series. Deep learning models [26, 24, 17, 4, 21] have demonstrated superior performance in MTS forecasting. In particular, Transformer-based models [34, 27, 35] have achieved great power in MTS, benefiting from its attention mechanism which can model the long-term interaction between different time points of sequences (cross-time). Graph Neural Networks (GNNs) [28, 16, 2, 19, 7] have also shown promising results for MTS forecasting, which can extract pre-defined or adaptive interaction between different variables (cross-variable).
However, a recent study [33] shows that a simple linear model dramatically outperformed many state-of-the-art (SOTA) models, and it inspires us to investigate the reasons why the existing cross-time
This work is done when Qihe Huang was an intern at YouTu Lab.
$\boxtimes$  Yang Wang and Zhengyang Zhou are corresponding authors.
and cross-variable interaction modeling fail to enhance prediction performance. By conducting a thorough analysis on real-world data, we observe that the presence of some unexpected noise (caused by humans, sensor distortion) may be responsible for it. In time dimension, as shown in Figure 1 (a), Transformer-based model heavily relies on the input sequences to generate the attention map, while its prediction may be susceptible to incidental noise, even some small fluctuation (i.e., noise) could easily lead to significant shifts on temporal dependencies. Our findings reveal that self-attention mechanism tend to assign high scores to outlier points in time series, resulting in spurious cross-time correlations. In variable dimension, cross-variable correlation exhibits a complex and dynamic evolution over time [18]. Despite the existence of underlying causal associations between variables, extracting cross-variable interaction is difficult due to the influence of noise interference. Additionally, as shown in Figure1 (b), we observe that such unexpected noise, which can be detected by the outlier detection algorithm [14], accounts for a high proportion in the time series.
(b)
Figure 1: Data analysis on the real-world dataset [31]. (a) Forecasting results of Transformer-based models suffering from unexpected noise. (b) Proportion of noise in the ETTh1, ETTm1, Traffic, and Weather datasets [27] detected by [14]. (c) Different levels of noise signals in multi-scale time series. (d) Homogeneous and heterogeneous relationships between variables.
Despite the non-negligible noise in time series, we can still discover the potential opportunities to confront noise challenges. (1) Cross-Scale Interaction. As shown on Figure 1 (c), by performing multi-scale extraction on the time series, we observe that different scales possess distinct levels of noise intensity, typically with coarser scales exhibiting lower noise intensity. Evidently, capturing dependencies across different scales enables the cross-time relationships to be robust against noise [15]. (2) Cross-Variable Interaction. As shown on Figure 1 (d), there is both homogeneity and heterogeneity for cross-variable interaction in real-world data [36]. In fact, the two relationships can both contribute to invariant connections during the temporal progression. Consequently, learning the invariant associations that contain both homogeneous and heterogeneous relationships among variables can boost its robustness to confront noise. Based on above analysis, it is still challenging to refine interaction in noisy MTS. The key obstacles can be summarized as follows: 1) How to capture cross-scale interaction that is not sensitive to unexpected input noise. 2) How to extract cross-variable relations between heterogeneous variables.
In this work, we propose CrossGNN, which is the first GNN solution to refine both cross-time and cross-variable interactions for MTS forecasting. To deal with the unexpected noise in time dimension, we firstly devise an adaptive multi-scale identifier (AMSI) to construct multi-scale time series with different noise levels. In time dimension, we propose Cross-Scale GNN, which is a temporal correlation graph, to model the dependency between different scales. The scales with clearer trends and weaker noise will be assigned with more edge weight. In variable dimension, we first introduce heterogeneous interaction modeling between variables into MTS forecasting and propose cross-variable GNN to utilize the homogeneity and heterogeneity between different variables with positive and negative edge weights. By focusing on edges with higher saliency scores and constraining those edges with lower scores at the same time, CrossGNN achieves linear time and space complexity (i.e.,  $O(L)$ ) with input sequence length  $L$ . The main contributions are summarized as follows:
- We conduct comprehensive studies on real-world MTS data and discover that the unexpected noise in time dimension and variable-wise heterogeneity between variables is not well handled by existing Transformer-based and GNN-based models.
- We propose a linear complexity CrossGNN model, which is the first GNN model to refine both cross-scale and cross-variable interaction for MTS forecasting.
1) To deal with the unexpected noise in time dimension, AMSI is leveraged to construct multi-scale time series with different noise level and a Cross-Scale GNN is proposed to capture the scales with clearer trend and weaker noise.
2) Cross-Variable GNN is designed to model the dynamic correlations between different variables. This the first model to introduce heterogeneous interaction modeling between variables into MTS forecasting.
- Extensive evaluation on 8 real-world MTS datasets demonstrates the effectiveness of CrossGNN. Specifically, CrossGNN achieved top-1 performance on 47 settings and top-2 on 9 settings when compared with 9 state-of-the-art models with varying prediction lengths.
Multivariate Time Series Forecasting. MTS forecasting has witnessed significant advancements due to the emergence of deep neural networks. These networks can be based on Convolutional Neural Network (CNN) [26, 23], Recurrent Neural Network (RNN) [5, 6], Transformer [11, 25, 27, 35, 9], or Graph Neural Network (GNN) [28, 32]. Generally, the primary emphasis of these studies lies in devising interactions between the temporal dimensions (cross-time) and the variable dimensions (cross-variable).
Cross-Time Interaction Modeling. Cross-time interaction modeling aims to capture correlations between different time points. Recently, CNN-based model TimesNet [26] transforms the time series into a two-dimensional matrix and uses a CNN-based backbone for feature extraction. RNN-based model LSTnet [6] utilizes the Long Short-Term Memory (LSTM) to model the temporal dependencies, but it may be limited by the inherent issue of gradient vanishing/exploding in RNNs. Transformer-based models benefit from its self-attention mechanism, enabling them to capture long-term cross-time dependency. AutoFormer [27] incorporates a decomposition mechanism that splits the input sequence into trend and seasonality, and integrate an auto-correlation module into the transformer to capture long-term cross-time dependency. FedFormer [35] leverages a frequency-enhanced decomposition mechanism while incorporating additional frequency information. However, despite the outstanding performance of Transformer-based methods, we observe that their self-attention mechanism is susceptible to unexpected noise, as shown on Figure 1(a). Based on these findings, we propose an innovative GNN-based method that constructs a cross-scale temporal graph to mitigate the impact of temporal noise on modeling cross-time correlations.
Cross-Variable Interaction Modeling. Cross-Variable Interaction is proved to be critical for MTS forecasting [34], and numerous works have employed Graph Neural Networks (GNNs) [22, 37, 30, 38] to capture cross-variable relationships. STGCN [32] firstly utilizes GNN to model the cross-variable dependency in traffic forecasting, which can effectively capture the dependency between different roads in pre-defined topology graphs. MTGNN [28] expands the utilization of GNN from spatiotemporal prediction to MTS forecasting, and proposes a straightforward method for computing adaptive cross-variable graph. On the other hand, the Transformer-based MTS prediction works have also recognized the potential of cross-variable interactions to enhance prediction performance, such as CrossFormer [34]. However, the cross-variable relationship is dynamic and can be greatly influenced by noise during the learning process. Given this, we refine the cross-variable relationship by decoupling homogeneity and heterogeneity in MTS, resulting in a noise-insensitive relationship during the temporal evolution.
In long-term multivariate time series (MTS) forecasting, the input comprises historical sequences across  $D$  variables denoted by  $\mathcal{X} = \{X_1^t,\dots,X_D^t\}_{t = 1}^L\in \mathbb{R}^{L\times D}$ , where  $L$  denotes the look-back window size and  $X_{i}^{t}$  denotes time series of the  $i_{th}$  variate at the  $t_{th}$  time step. The objective of MTS forecasting is to predict future time series denoted by  $\hat{\mathcal{X}} = \{\hat{X}_1^t,\dots,\hat{X}_D^t\}_{t = L + 1}^{L + T}\in \mathbb{R}^{T\times D}$  based on  $\mathcal{X}$ , where  $T$  represents the prediction time steps and  $T\gg 1$ . The detailed structure of CrossGNN is illustrated in Figure 2. We firstly employ an adaptive multi-scale identifier (AMSI) to generate multi-scale time series and reduce noise on coarse scale. Then, we construct scale-sensitive and trend-aware temporal graph to extract cross-scale interaction. We perform cross-variable aggregation via
(a) Adaptive multi scale identifier
Figure 2: CrossGNN architecture. (a) Adaptive multi-scale identifier (AMSI) is devised to extract the multi-scale MTS  $\mathcal{X}'$  from the input  $\mathcal{X}$ . (b) Cross-Scale GNN facilitates cross-time interaction within and across different scales, while Cross-Variable GNN models cross-variable interaction for both homogeneous and heterogeneous relationships. (c) Direct multi-step (DMS) forecasting is leveraged to predict the future time series based on two MLPs.
(b) Cross interaction layer
(c) Forecasting
Channel dimension linear project
Time dimension linear project
modeling homogeneous and heterogeneous relationships between variables. Finally, direct multi-step (DMS) forecasting is adopted in decoder to predict future time series.
3.1 Adaptive Multi-Scale Identifier
The adaptive multi-scale identifier (AMSI) is designed to capture the different scales of MTS from coarse to fine, and reduce the unexpected noise on the coarse scale. Technically, we utilize Fast Fourier Transform (FFT) to analyze the time series in the frequency domain and calculate the potential periods of the time series, inspired by [26]. We compute the amplitude of each time series at various frequencies using FFT and subsequently average the amplitude of  $\mathcal{X}$  across variable dimension:
$$
A = \operatorname {A v g} _ {i = 1} ^ {D} (\operatorname {A m p} (\operatorname {F F T} (\mathcal {X}))), \tag {1}
$$
where  $\mathrm{Amp}(\cdot)$  is the calculation of amplitude,  $\mathrm{FFT}(\cdot)$  is the calculation of FFT,  $A\in \mathbb{R}^{L}$  represents the calculated amplitude of each frequency, which is averaged from  $D$  variables by  $\mathrm{Avg}(\cdot)$ . We choose the frequencies  $\{f_1,f_2,\dots ,f_S\}$  which correspond to the Top-S amplitude values:
$$
\left\{f _ {1}, \dots , f _ {S} \right\} = \arg \operatorname {T o p - S} (A), \tag {2}
$$
where  $\arg \operatorname{Top-S}(\cdot)$  picks out the  $S$  frequency values with the highest amplitude from  $A$ . The period lengths  $\{p_1, p_2, \dots, p_S\}$  are calculated by the selected frequencies as follows:
$$
p _ {s} = \left\lceil \frac {L}{f _ {s}} \right\rceil , s \in \{1, \dots , S \}. \tag {3}
$$
Then,  $\mathrm{AvgPool}(\cdot)$  with kernel size  $p_s$  and stride  $p_s$ ,  $s \in \{1, 2, \dots, S\}$ , are applied to the MTS  $\mathcal{X} \in \mathbb{R}^{L \times D}$  in the time dimension to capture the MTS at  $s$ -th scales:
$$
\mathcal {X} _ {s} = \operatorname {A v g P o o l} (\mathcal {X}) _ {\text {k e r n e l} = \text {s t r i d e} = p _ {s}}, \tag {4}
$$
where  $\operatorname{AvgPool}(\cdot)$  downsamples time series to obtain  $\mathcal{X}_s \in \mathbb{R}^{L(s) \times D}$ ,  $L(s) = \left\lfloor \frac{L}{p_s} \right\rfloor$  is the time series length in the  $s$ -th scale,  $\lfloor \cdot \rfloor$  is the operation of rounding down. We concatenate the captured different
scales in the time dimension and obtain a period-wise multi-scale MTS  $\mathcal{X}' \in \mathbb{R}^{L' \times D}$  as the output of AMSI,  $L' = \sum_{s=1}^{S} L(s)$  is the sum of lengths across all scales:
$$
\mathcal {X} ^ {\prime} = \operatorname {C o n c a t} \left(\mathcal {X} _ {1}, \mathcal {X} _ {2}, \dots , \mathcal {X} _ {S}\right). \tag {5}
$$
Here, we employ an expansion dimension strategy (using an MLP), to create an embedding for each time step at different scale. This strategy is inherited from MTGNN [28], aiming to enhance the local semantics at each time step and positively impact subsequent cross-scale and cross-variable interactions. The shape of  $X'$  is finally expanded as  $\mathbb{R}^{L \times D \times C}$ .
3.2 Cross-Scale GNN
The Cross-Scale GNN is designed to utilize the interaction of the multi-scale MTS  $\mathcal{X}'$  and extract the scales with clearer association and weaker noise. The cross-scale graph in the time dimension is represented as  $G^{scale} = (V^{scale}, E^{scale})$ .  $V^{scale} = \{v_1^{scale}, v_2^{scale}, \dots, v_{L'}^{scale}\}$  is the time nodes set in all time scales, where  $v_i^{scale}$  is the  $i$ -th time node.  $E^{scale} \in \mathbb{R}^{L' \times L'}$  assigns the correlation weights between each time node, each element in  $E^{scale}$  meaning the correlation weight between two time nodes (inter-scale or intra-scale). The key purpose of Cross-scale GNN is to learn the cross-scale temporal correlation weights  $E^{scale}$  that is insensitive to noise interference. To diminish the effect of noise on correlation weights, we maintain the independence by initializing  $E^{scale}$  with the production of two learnable vectors  $vec_1^{scale}$  and  $vec_2^{scale}$ , ensuring  $E^{scale}$  not affected by noise hidden in input:
$$
E ^ {\text {s c a l e}} = \operatorname {S o f t m a x} \left(\operatorname {R e L U} \left(\operatorname {v e c} _ {1} ^ {\text {s c a l e}} \times \operatorname {v e c} _ {2} ^ {\text {s c a l e}}\right)\right), \tag {6}
$$
where  $\mathrm{ReLU}(\cdot)$  is the active function regularizing the weight matrix so that each element is positive, and  $\mathrm{Softmax}(\cdot)$  is the operation to ensure the weights of all nodes correlated to a particular time node sum to 1.
Scale-sensitive Restriction. We consider that for any time node, the number of its correlated time nodes on the fine scale should be more than the number on the coarse scale. For any time node  $v_{i}^{scale}$ , the number of its correlated nodes at  $s$ -th scale is restricted to  $k_{s} = \lceil \frac{K}{p_{s}} \rceil$ , where  $p_{s}$  is the period length of  $s$ -th scale,  $\lceil \cdot \rceil$  denotes the ceiling function,  $K$  is a constant. This ensures that finer-scale time series contribute more temporal node associations. The neighbor time node set at  $s$ -th scale (i.e. the correlated time nodes at  $s$ -th scale) of  $v_{i}^{scale}$  is denoted as:
$$
\mathcal {N} _ {s} ^ {\text {s c a l e}} \left(v _ {i} ^ {\text {s c a l e}}\right) = \arg \operatorname {T o p - k} _ {\mathrm {s}} \left(E _ {s} ^ {\text {s c a l e}} \left(v _ {i} ^ {\text {s c a l e}}\right)\right), \tag {7}
$$
where  $\arg \mathrm{Top - k_s}(\cdot)$  is an operation to extract  $k_{s}$  nodes with highest correlation weight,  $E_{s}^{scale}(v_{i}^{scale})\in \mathbb{R}^{L(s)}$  is the correlation weight of time node  $v_{i}^{scale}$  at  $s$ -th scale. In this way, the number of neighboring nodes at different scales can be restricted based on the correlation weight matrix  $E^{scale}$ .
Trend-aware Selection. To ensure the temporal trends can be captured, we preserve the associations between a time node and its both preceding and succeeding nodes. Denote  $\mathcal{N}^{trend}(v_i^{scale})$  as the trend neighbor set of time node  $v_{i}^{scale}$ , which can be defined as follows:
$$
\mathcal {N} ^ {t r e n d} \left(v _ {i} ^ {s c a l e}\right) = \left\{v _ {j} ^ {s c a l e} \mid | i - j | \leq 1, s c a l e \left(v _ {i} ^ {s c a l e}\right) = s c a l e \left(v _ {j} ^ {s c a l e}\right) \right\}, \tag {8}
$$
where  $\text{scale}(\cdot)$  provides the scale of a time node. The trend neighbor set of  $v_{i}^{\text{scale}}$  consists of its adjacent time nodes (i.e.,  $|i - j| \leq 1$ ) which share the same scale. This gives the ability to preserve temporal trends in the cross-scale correlation graph.
Correlation Weight Re-normalization. Denote  $\mathcal{N}(v_i) = \mathcal{N}^{scale}(v_i) \cup \mathcal{N}^{trend}(v_i)$  as the selected neighbor set of time nodes  $v_i^{scale}$ . The final correlation weight is re-normalized as follows:
$$
E ^ {\text {s c a l e}} [ i, j ] = \left\{ \begin{array}{l l} \frac {E ^ {\text {s c a l e}} [ i , j ]}{\sum_ {v _ {m} \in \mathcal {N} \left(v _ {i}\right)} E ^ {\text {s c a l e}} [ i , m ]}, & \text {i f} v _ {j} \in \mathcal {N} \left(v _ {i}\right), \\ 0, & \text {o t h e r w i s e}, \end{array} \right. \tag {9}
$$
where  $E^{scale}[i,j]$  is the correlation weight between  $v_{i}^{scale}$  and  $v_{j}^{scale}$ . This step filters out non-significant correlation and preserves a restricted set of neighboring nodes  $\mathcal{N}(v_{i}^{scale})$  to each node. Also, re-normalization is applied to the retained correlation. Then, cross-scale correlation graph between time nodes at different scales is constructed.
Cross-scale interaction. After obtaining the cross-scale temporal correlation graph, we do cross-scale interaction in the time dimension based on GNN, and the information propagation process will be stacked for  $N$  layers:
$$
\mathcal {H N} _ {i,:} ^ {\text {t i m e}, N} = (\sum_ {s = 1} ^ {S} \sum_ {v _ {j} \in \mathcal {N} _ {s} ^ {\text {s c a l e}} (v _ {i})} E [ i, j ] \cdot \mathcal {H} _ {j,:} ^ {\text {t i m e}, N - 1}) + (\sum_ {v _ {m} \in \mathcal {N} ^ {\text {t r e n d}} (v _ {i})} E [ i, m ] \cdot \mathcal {H} _ {m,:} ^ {\text {t i m e}, N - 1})
$$
$$
\mathcal {H} _ {i,:} ^ {\text {t i m e}, N} = \sigma (\operatorname {C o n c a t} (\mathcal {H N} _ {i,:} ^ {\text {t i m e}, N}, \mathcal {H} _ {i,:} ^ {\text {t i m e}, N - 1}) \cdot W)
$$
$$
\mathcal {H} _ {i,:} ^ {\text {t i m e}, N} = \mathcal {H} _ {i,:} ^ {\text {t i m e}, N} / \left\| \mathcal {H} _ {i,:} ^ {\text {t i m e}, N} \right\| _ {2} \tag {10}
$$
where  $\sigma$  is the activation function;  $W$  is the learnable matrix;  $\mathcal{H}^{time}$  is the time node feature;  $\mathcal{H}\mathcal{N}^{time}$  is the aggregation of neighbor time node feature.  $\mathcal{HN}_{i,:}^{time,N}$  aggregates the neighboring node feature from the previous layer correlated to the time nodes of  $v_{i}^{scale}$ . Then  $\mathcal{H}_{i,:}^{time,N}$  is updated by the aggregated time node feature  $\mathcal{HN}_{i,:}^{time,N}$  and its previous layer feature  $\mathcal{H}_{i,:}^{time,N-1}$ . Finally, the normalized  $\mathcal{H}_{i,:}^{time,N}$  at  $N$ -th layer is the output of Cross-Scale GNN.
3.3 Cross-Variable GNN
Cross-Variable GNN is designed to extract the invariant correlation consisting of homogeneous and heterogeneous relationship. In variable dimension, the cross-variable graph is represented as  $G^{var} = (V^{var}, E^{var})$ .  $V^{var} = \{v_{1}^{var}, v_{2}^{var}, \dots, v_{D}^{var}\}$  is the variable node set, where  $D$  is the number of variables, and  $v_{i}^{var}$  is the  $i$ -th variable node.  $E^{var}$  is the variable correlation weight matrix, each element in  $E^{var}$  meaning the correlation weight between two variables.  $E^{var}$  is initialized by production of two latent vectors  $vec_{1}^{var}$  and  $vec_{2}^{var}$ :
$$
E ^ {v a r} = \operatorname {S o f t m a x} \left(\operatorname {R e L U} \left(v e c _ {1} ^ {v a r} \times v e c _ {2} ^ {v a r}\right)\right) \tag {11}
$$
Heterogeneity Disentanglement. Specifically, we select the nodes with  $K_{+}^{var}$  highest correlation weight as positive neighbors with homogeneous connections, and take the nodes with  $K_{-}^{var}$  lowest correlation score as negative neighbors with heterogeneous connections. Denote  $E^{var}(v_i^{var})$  as the correlation weight of  $D$  variable nodes related to  $v_{i}^{var}$ . For variable  $v_{i}$ , its two decoupled neighbor sets can be represented as  $\mathcal{N}_{-}^{var}(v_{i}) = \mathrm{Bottom - K}_{-}^{\mathrm{var}}(E^{var}(v_{i}))$  and  $\mathcal{N}_{+}^{var}(v_{i}) = \mathrm{Top - K}_{+}^{\mathrm{var}}(E^{var}(v_{i}))$ , respectively.
Correlation Weight Re-normalization. The corresponding homogeneous and heterogeneous correlation weights are derived as follows:
$$
E ^ {v a r} [ i, j ] = \left\{ \begin{array}{l l} - \frac {\frac {1}{E ^ {v a r} [ i , j ]}}{\sum_ {v _ {k} \in \mathcal {N} _ {+} ^ {v a r} (v _ {i})} \frac {1}{E ^ {v a r} [ i , k ]}}, & \text {i f} v _ {j} \in \mathcal {N} _ {-} ^ {v a r} (v _ {i}), \\ \frac {E ^ {v a r} [ i , j ]}{\sum_ {v _ {k} \in \mathcal {N} _ {+} ^ {v a r} (v _ {i})} E ^ {v a r} [ i , k ]}, & \text {i f} v _ {j} \in \mathcal {N} _ {+} ^ {v a r} (v _ {i}), \\ 0, & \text {o t h e r w i s e} \end{array} \right. \tag {12}
$$
This process filters out edges other than homogeneous and heterogeneous edges. The weights of homogeneous edges are positively correlated with their correlation scores, while the weights of heterogeneous edges are negatively correlated with their correlation scores. Additionally, separate renormalization is applied to the weights of homogeneous edges and heterogeneous edges, respectively. Then, the cross-variable graph is constructed with disentangled homogeneous and heterogeneous correlations.