SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Kun Yi $^{1}$ , Qi Zhang $^{2}$ , Wei Fan $^{3}$ , Hui He $^{1}$ , Liang Hu $^{2}$ , Pengyang Wang $^{4}$ , Ning An $^{5}$ , Longbing Cao $^{6}$ , Zhendong Niu $^{1*}$
<sup>1</sup>Beijing Institute of Technology, <sup>2</sup>Tongji University, <sup>3</sup>University of Oxford
<sup>4</sup>University of Macau, <sup>5</sup>HeFei University of Technology, <sup>6</sup>Macquarie University {yikun, hehui617, zniu} @bit.edu.cn, zhangqi_cs@tongji.edu.cn, weifan.oxford@gmail.com railmilk@gmail.com, pywang@um.edu.mo, ning.g.an@acm.org, longbing.cao@mq.edu.au
Multivariate time series (MTS) forecasting has shown great importance in numerous industries. Current state-of-the-art graph neural network (GNN)-based forecasting methods usually require both graph networks (e.g., GCN) and temporal networks (e.g., LSTM) to capture inter-series (spatial) dynamics and intra-series (temporal) dependencies, respectively. However, the uncertain compatibility of the two networks puts an extra burden on handcrafted model designs. Moreover, the separate spatial and temporal modeling naturally violates the unified spatiotemporal inter-dependencies in real world, which largely hinders the forecasting performance. To overcome these problems, we explore an interesting direction of directly applying graph networks and rethink MTS forecasting from a pure graph perspective. We first define a novel data structure, hypervariate graph, which regards each series value (regardless of variates or timestamps) as a graph node, and represents sliding windows as space-time fully-connected graphs. This perspective considers spatiotemporal dynamics unitedly and reformulates classic MTS forecasting into the predictions on hypervariate graphs. Then, we propose a novel architecture Fourier Graph Neural Network (FourierGNN) by stacking our proposed Fourier Graph Operator (FGO) to perform matrix multiplications in Fourier space. FourierGNN accommodates adequate expressiveness and achieves much lower complexity, which can effectively and efficiently accomplish the forecasting. Besides, our theoretical analysis reveals FGO's equivalence to graph convolutions in the time domain, which further verifies the validity of FourierGNN. Extensive experiments on seven datasets have demonstrated our superior performance with higher efficiency and fewer parameters compared with state-of-the-art methods. Code is available at this repository: https://github.com/aikunyi/FourierGNN.
Multivariate time series (MTS) forecasting plays an important role in numerous real-world scenarios, such as traffic flow prediction in transportation systems [1, 2], temperature estimation in weather forecasting [3, 4], and electricity consumption planning in the energy market [5, 6], etc. In MTS forecasting, the core challenge is to model intra-series (temporal) dependencies and simultaneously capture inter-series (spatial) correlations. Existing literature has primarily focused on the temporal modeling and proposed several forecasting architectures, including Recurrent Neural Network (RNN)-based methods (e.g., DeepAR [7]), Convolution Neural Network (CNN)-based methods (e.g., Temporal Convolution Network [8]) and more recent Transformer-based methods (e.g., Informer [9]
and Autoformer [4]). In addition, another branch of MTS forecasting methods has been developed to not only model temporal dependencies but also places emphasis on spatial correlations. The most representative methods are the emerging Graph Neural Network (GNN)-based approaches [10, 2, 11] that have achieved state-of-the-art performance in the MTS forecasting task.
Previous GNN-based forecasting methods (e.g., STGCN [12] and TAMP-S2GC nets [11]) heavily rely on a pre-defined graph structure to specify the spatial correlations, which as a matter of fact cannot capture the spatial dynamics, i.e., the time-evolving spatial correlation patterns. Later advanced approaches (e.g., StemGNN [10], MTGNN [13], AGCRN [2]) can automatically learn inter-series correlations and accordingly model spatial dynamics without pre-defined priors, but almost all of them are designed by stacking graph networks (e.g., GCN and GAT) to capture spatial dynamics and temporal networks (e.g., LSTM and GRU) to capture temporal dependencies. However, the uncertain compatibility of the graph networks and the temporal networks puts extra burden on handcrafted model designs, which hinders the forecasting performance. Moreover, the respective modeling for the two networks separately learn spatial/temporal correlations, which naturally violate the real-world unified spatiotemporal inter-dependencies. In this paper, we explore an opposite direction of directly applying graph networks for forecasting and investigate an interesting question: can pure graph networks capture spatial dynamics and temporal dependencies even without temporal networks?
To answer this question, we rethink the MTS forecasting task from a pure graph perspective. We start with building a new data structure, hypervariate graph, to represent time series with a united view of spatial/temporal dynamics. The core idea of the hypervariate graph is to construct a space-time fully-connected structure. Specifically, given a multivariate time series window (say input window)  $X_{t} \in \mathbb{R}^{N \times T}$  at timestamp  $t$ , where  $N$  is the number of series (variates) and  $T$  is the length of input window, we construct a corresponding hypervariate graph structure represented as  $\mathcal{G}_t^T = (X_t^T, A_t^T)$ , which is initialized as a fully-connected graph of  $NT$  nodes with adjacency matrix  $A_t^T \in \mathbb{R}^{NT \times NT}$ .
and node features  $X_{t}^{T} \in \mathbb{R}^{NT \times 1}$  by regarding each value  $x_{t}^{(n)} \in \mathbb{R}^{1}$  (variate  $n$  at step  $t$ ) of input window as a distinct node of a hypervariate graph. Such a special structure design formulates both intra- and inter-series correlations of multivariate series as pure node-node dependencies in the hypervariate graph. Different from classic formulations that make spatial-correlated graphs and learn dynamics in a two-stage (spatial and temporal) process [13], our perspective views spatiotemporal correlations as a whole. It abandons the uncertain compatibility of spatial/temporal modeling, constructs adaptive space-time inter-dependencies, and brings up higher-resolution fusion across multiple variates and timestamps in MTS forecasting.
Then, with such a graph structure, the multivariate forecasting can be originally formulated into the predictions on the hypervariate graph. However, the node number of the hypervariate graph increase with the number of series  $(N)$  and the window length  $(T)$ , leading to a graph of large order and size. This could make classic graph networks (e.g., GCN [14], GAT [15]) computationally expensive (usually with quadratic complexity) and suffer from optimization difficulty in obtaining accurate node representations [16]. To this end, we propose a novel architecture, Fourier Graph Neural Network (FourierGNN), for MTS forecasting from a pure graph perspective. Specifically, FourierGNN is built upon our proposed Fourier Graph Operator (FGO), which as a replacement of classic graph operation units (e.g., convolutions), performs matrix multiplications in Fourier space of graphs. By stacking FGO layers in Fourier space, FourierGNN can accommodate adequate learning expressiveness and in the mean time achieve much lower complexity (Log-linear complexity), which thus can effectively and efficiently accomplish MTS forecasting. Besides, we present theoretical analysis to demonstrate that the FGO is equivalent to graph convolutions in the time domain, which further explains the validity of FourierGNN.
Finally, we perform extensive experiments on seven real-world benchmarks. Experimental results demonstrate that FourierGNN achieves an average of more than  $10\%$  improvement in accuracy compared with state-of-the-art methods. In addition, FourierGNN achieves higher forecasting efficiency, which has about  $14.6\%$  less costs in training time and  $20\%$  less parameter volumes, compared with most lightweight GNN-based forecasting methods.
Graph Neural Networks for Multivariate Time Series Forecasting Multivariate time series (MTS) have embraced GNN due to their best capability of modeling structural dependencies between variates [17, 2, 13, 12, 11, 18, 19]. Most of these models, such as STGCN [12], DCRNN [18],
and TAMP-S2GC nets [11], require a pre-defined graph structure which is usually unknown in most cases. For this limitation, some studies enable to automatically learn the graphs by the inter-series correlations, e.g., by node similarity [20, 2, 17] or self-attention mechanism [10]. However, these methods always adopt a graph network for spatial correlations and a temporal network for temporal dependencies separately [20, 17, 10, 2]. For example, AGCRN [2] use a GCN [14] and a GRU [21], GraphWaveNet [17] use a GCN and a TCN [8], etc. In this paper, we propose an unified spatiotemporal formulation with pure graph networks for MTS forecasting.
Multivariate Time Series Forecasting with Fourier Transform Recently, many MTS forecasting models have integrated the Fourier theory into deep neural networks [22, 23]. For instance, SFM [24] decomposes the hidden state of LSTM into multiple frequencies by Discrete Fourier Transform (DFT). mWDN [25] decomposes the time series into multilevel sub-series by discrete wavelet decomposition (DWT) and feeds them to LSTM network. ATFN [26] proposes a Discrete Fourier Transform-based block to capture dynamic and complicated periodic patterns of time series data. FEDformer [27] proposes Discrete Fourier Transform-based attention mechanism with low-rank approximation in frequency. While these models only capture temporal dependencies with Fourier Transform, StemGNN [10] takes the advantages of both spatial correlations and temporal dependencies in the spectral domain by utilizing Graph Fourier Transform (GFT) to perform graph convolutions and Discrete Fourier Transform (DFT) to calculate the series relationships.
Given the multivariate time series input, i.e., the lookback window  $X_{t} = [\pmb{x}_{t - T + 1},\dots,\pmb{x}_{t}]\in \mathbb{R}^{N\times T}$  at timestamps  $t$  with the number of series (variates)  $N$  and the lookback window size  $T$ , where  $\pmb{x}_t\in \mathbb{R}^N$  denotes the multivariate values of  $N$  series at timestamp  $t$ . Then, the multivariate time series forecasting task is to predict the next  $\tau$  timestamps  $Y_{t} = [\pmb{x}_{t + 1},\dots,\pmb{x}_{t + \tau}]\in \mathbb{R}^{N\times \tau}$  based on the historical  $T$  observations  $X_{t} = [\pmb{x}_{t - T + 1},\dots,\pmb{x}_{t}]$ . The forecasting process can be given by:
$$
\hat {Y} _ {t} := F _ {\theta} \left(X _ {t}\right) = F _ {\theta} \left(\left[ \boldsymbol {x} _ {t - T + 1}, \dots , \boldsymbol {x} _ {t} \right]\right) \tag {1}
$$
where  $\hat{Y}_t$  are the predictions corresponding to the ground truth  $Y_t$ . The forecasting function is denoted as  $F_{\theta}$  parameterized by  $\theta$ . In practice, many MTS forecasting models usually leverage a graph network (assume parameterized by  $\theta_g$ ) to learn the spatial dynamics and a temporal network (assume parameterized by  $\theta_t$ ) to learn the temporal dependencies, respectively [17, 10, 2, 13, 11]. Thus, the original definition of Equation (1) can be rewritten to:
$$
\hat {Y} _ {t} := F _ {\theta_ {g}, \theta_ {t}} (X _ {t}) = F _ {\theta_ {g}, \theta_ {t}} ([ \boldsymbol {x} _ {t - T + 1}, \dots , \boldsymbol {x} _ {t} ]) \tag {2}
$$
where original parameters  $\theta$  are exposed to the parameters of the graph network  $\theta_{g}$  and the temporal network  $\theta_{t}$  to make prediction based on the learned spatial-temporal dependencies.
In this section, we elaborate on our proposed framework: First, we start with our pure graph formulation with a novel hypervariate graph structure for MTS forecasting in Section 4.1. Then, we illustrate the proposed neural architecture, Fourier Graph Neural Network (FourierGNN), for this formulation in Section 4.2. Besides, we theoretically analyze FourierGNN to demonstrate its architecture validity, and also conduct complexity analysis to show its efficiency. Finally, we introduce certain inductive bias to instantiate FourierGNN for MTS forecasting in Section 4.3.
4.1 The Pure Graph Formulation
To overcome the uncertain compatibility of the graph network and the temporal network as aforementioned in Section 1, and learn the united spatiotemporal dynamics, we propose a pure graph formulation that refines Equation (2) by a novel data structure, hypervariate graph, for time series.
Definition 1 (Hypervariate Graph). Given a multivariate time series window as input  $X_{t} \in \mathbb{R}^{N \times T}$  of  $N$  variates at timestamp  $t$ , we construct a hypervariate graph of  $NT$  nodes,  $\mathcal{G}_t = (X_t^\mathcal{G}, A_t^\mathcal{G})$ , by regarding each element of  $X_{t}$  as one node of  $\mathcal{G}_t$  such that  $X_{t}^{\mathcal{G}} \in \mathbb{R}^{NT \times 1}$  stands for the node feature and  $A_{t}^{\mathcal{G}} \in \mathbb{R}^{NT \times NT}$  is the adjacency matrix initialized to make  $\mathcal{G}_t$  as a fully-connected graph.
Since the prior graph structure is usually unknown in most multivariate time series scenarios [10, 2, 13], and the elements of  $X_{t}$  are spatially or temporally correlated with each other because of time lag effect [28], we assume all nodes in the hypervariate graph  $\mathcal{G}_t$  are fully-connected. The hypervariate graph  $\mathcal{G}_t$  contains  $NT$  nodes representing the values of each variate at each timestamp in  $X_{t}$ , which can learn a high-resolution representation across timestamps and variates (more explanations of the hypervariate graph can be seen in Appendix C.1). We present an example hypervariate graph of three time series in Figure 1. Thus, with such a data structure, we can reformulate the multivariate
Figure 1: Illustration of a hypervariate graph with three time series. Each value in the input window is considered as a node of the graph.
time series forecasting task into the predictions on the hypervariate graphs, and accordingly rewrite Equation (2) into:
$$
\hat {Y} _ {t} := F _ {\theta_ {\mathcal {G}}} \left(X _ {t} ^ {\mathcal {G}}, A _ {t} ^ {\mathcal {G}}\right) \tag {3}
$$
where  $\theta_{\mathcal{G}}$  stands for the network parameters for hypervariate graphs. With such a formulation, we can view the spatial dynamics and the temporal dependencies from an united perspective, which benefits modeling the real-world spatiotemporal inter-dependencies.
4.2 FourierGNN
Though the pure graph formulation can enhance spatiotemporal modeling, the order and size of hypervariate graphs increase with the number of variates  $N$  and the size of window  $T$ , which makes classic graph networks (e.g., GCN [14] and GAT [15]) computationally expensive (usually quadratic complexity) and suffer from optimization difficulty in obtaining accurate hidden node representations [16]. In this regard, we propose an efficient and effective method, FourierGNN, for the pure graph formulation. The main architecture of FourierGNN is built upon our proposed Fourier Graph Operator (FGO), a learnable network layer in Fourier space, which is detailed as follows.
Definition 2 (Fourier Graph Operator). Given a graph  $G = (X, A)$  with node features  $X \in \mathbb{R}^{n \times d}$  and the adjacency matrix  $A \in \mathbb{R}^{n \times n}$ , where  $n$  is the number of nodes and  $d$  is the number of features, we introduce a weight matrix  $W \in \mathbb{R}^{d \times d}$  to acquire a tailored Green's kernel  $\kappa : [n] \times [n] \to \mathbb{R}^{d \times d}$  with  $\kappa[i, j] := A_{ij} \circ W$  and  $\kappa[i, j] = \kappa[i - j]$ . We define  $S_{A,W} := \mathcal{F}(\kappa) \in \mathbb{C}^{n \times d \times d}$  as a Fourier Graph Operator (FGO), where  $\mathcal{F}$  denotes Discrete Fourier Transform (DFT).
According to the convolution theorem [29] (see Appendix B), we can write the multiplication between  $\mathcal{F}(X)$  and FGO  $S_{A,W}$  in Fourier space as:
$$
\mathcal {F} (X) \mathcal {F} (\kappa) = \mathcal {F} ((X * \kappa) [ i ]) = \mathcal {F} \left(\sum_ {j = 1} ^ {n} X [ j ] \kappa [ i - j ]\right) = \mathcal {F} \left(\sum_ {j = 1} ^ {n} X [ j ] \kappa [ i, j ]\right), \quad \forall i \in [ n ] \tag {4}
$$
where  $(X*\kappa)[i]$  denotes the convolution of  $X$  and  $\kappa$ . As defined  $\kappa[i,j] = A_{ij} \circ W$ , it yields  $\sum_{j=1}^{n} X[j]\kappa[i,j] = \sum_{j=1}^{n} A_{ij}X[j]W = AXW$ . Accordingly, we can get the convolution equation:
$$
\mathcal {F} (X) \mathcal {S} _ {A, W} = \mathcal {F} (A X W). \tag {5}
$$
In particular, turning to our case of the fully-connected hypervariate graph, we can adopt a  $n$ -invariant FGO  $\mathcal{S} \in \mathbb{C}^{d \times d}$  that has a computationally low cost compared to previous  $\mathbb{C}^{n \times d \times d}$ . We provide more details and explanations in Appendix C.2.
From Equation (5), we can observe that performing the multiplication between  $\mathcal{F}(X)$  and FGO  $S$  in Fourier space corresponds to a graph shift operation (i.e., a graph convolution) in the time domain [20]. Since the multiplications in Fourier space  $(\mathcal{O}(n))$  have much lower complexity than the above shift operations  $(\mathcal{O}(n^2))$  in the time domain (See Complexity Analysis below), it motivates us to develop a highly efficient graph neural network in Fourier space.
To this end, we propose the Fourier Graph Neural Networks (FourierGNN) based on FGO. Specifically, by stacking multiple layers of FGOs, we can define the  $K$ -layer Fourier graph neural networks given a graph  $G = (X,A)$  with node features  $X \in \mathbb{R}^{n \times d}$  and the adjacency matrix  $A \in \mathbb{R}^{n \times n}$  as:
$$
\operatorname {F o u r i e r G N N} (X, A) := \sum_ {k = 0} ^ {K} \sigma (\mathcal {F} (X) \mathcal {S} _ {0: k} + b _ {k}), \quad \mathcal {S} _ {0: k} = \prod_ {i = 0} ^ {k} \mathcal {S} _ {i}. \tag {6}
$$
Herein,  $S_{k}$  is the FGO in the  $k$ -th layer, satisfying  $\mathcal{F}(X)\mathcal{S}_k = \mathcal{F}(A_kXW_k)$  with  $W_{k} \in \mathbb{R}^{d \times d}$  being the weights and  $A_{k} \in \mathbb{R}^{n \times n}$  corresponding to the  $k$ -th adjacency matrix sharing the same sparsity pattern of  $A$ , and  $b_{k} \in \mathbb{C}^{d}$  are the complex-valued biases parameters;  $\mathcal{F}$  stands for Discrete Fourier Transform;  $\sigma$  is the activation function. In particular,  $S_{0}, W_{0}, A_{0}$  are the identity matrix, and we adopt identical activation at  $k = 0$  to obtain residual  $\mathcal{F}(X)$ . All operations in FourierGNN are performed in Fourier space. Thus, all parameters, i.e.,  $\{\mathcal{S}_k, b_k\}_{k=1}^K$ , are complex numbers.
The core operation of FourierGNN is the summation of recursive multiplications with nonlinear activation functions. Specifically, the recursive multiplications between  $\mathcal{F}(X)$  and  $S$ , i.e.,  $\mathcal{F}(X)S_{0:k}$ , are equivalent to the multi-order convolutions on the graph structure (see Theoretical Analysis below). Nonlinear activation functions  $\sigma$  are introduced to address the capability limitations of modeling nonlinear information diffusion on graphs in the summation.
Theoretical Analysis We theoretically analyze the effectiveness and interpretability of FourierGNN and verify the validity of its architecture. For convenience, we exclude the non-linear activation function  $\sigma$  and learnable bias parameters  $b$  from Equation (6), and focus on  $\mathcal{F}(X)\mathcal{S}_{0:k}$ .
Proposition 1. Given a graph  $G = (X, A)$  with node features  $X \in \mathbb{R}^{n \times d}$  and adjacency matrix  $A \in \mathbb{R}^{n \times n}$ , the recursive multiplication of FGOs in Fourier space is equivalent to multi-order convolutions in the time domain:
$$
\mathcal {F} ^ {- 1} \left(\mathcal {F} (X) \mathcal {S} _ {0: k}\right) = A _ {k: 0} X W _ {0: k}, \quad \mathcal {S} _ {0: k} = \prod_ {i = 0} ^ {k} \mathcal {S} _ {i}, A _ {k: 0} = \prod_ {i = k} ^ {0} A _ {i}, W _ {0: k} = \prod_ {i = 0} ^ {k} W _ {i} \tag {7}
$$
where  $A_0, S_0, W_0$  are the identity matrix,  $A_k \in \mathbb{R}^{n \times n}$  corresponds to the  $k$ -th diffusion step sharing the same sparsity pattern of  $A$ ,  $W_k \in \mathbb{R}^{d \times d}$  is the  $k$ -th weight matrix,  $S_k \in \mathbb{C}^{d \times d}$  is the  $k$ -th FGO satisfying  $\mathcal{F}(A_k X W_k) = \mathcal{F}(X) S_k$ , and  $\mathcal{F}$  and  $\mathcal{F}^{-1}$  denote DFT and its inverse, respectively.
In the time domain, operation  $A_{k:0}XW_{0:k}$  adopts different weights  $W_{k} \in \mathbb{R}^{d \times d}$  to weigh the information of different neighbors in different diffusion orders, beneficial to capture the extensive dependencies on graphs [20, 30, 31]. This indicates FourierGNN is expressive in modeling the complex correlations among graph nodes, i.e., spatiotemporal dependencies in the hypervariate graph. The proof of Proposition 1 and more explanations of FourierGNN are provided in Appendix C.3.
Complexity Analysis The time complexity of  $\mathcal{F}(X)\mathcal{S}$  is  $\mathcal{O}(nd\log n + nd^2)$ , which includes the Discrete Fourier Transform (DFT), the Inverse Discrete Fourier Transform (IDFT), and the matrix multiplication in the Fourier space. Comparatively, the time complexity of the equivalent operations of  $\mathcal{F}(X)\mathcal{S}$  in the time domain, i.e.,  $AXW$ , is  $\mathcal{O}(n^2d + nd^2)$ . Then, as a  $K$ -order summation of a recursive multiplication of  $\mathcal{F}(X)\mathcal{S}$ , FourierGNN, achieves the time complexity of  $\mathcal{O}(nd\log n + Knd^2)$ , including DFT and IDFT, and the recursive multiplication of FGOs. Overall, the Log-linear  $\mathcal{O}(n\log n)$  complexity makes FourierGNN much more efficient.
FourierGNN vs Other Graph Networks We analyze the connection and difference between our FourierGNN with GCN [14] and GAT [15]. From the complexity perspective, FourierGNN with log-linear complexity shows much higher efficiency than GCN and GAT. Regarding the network architecture, we analyze them from two main perspectives: (1) Domain. GAT implements operations in the time domain, while GCN and FourierGNN are in Fourier space. However, GCN achieves the transformation through the Graph Fourier Transform (GFT), whereas FourierGNN utilizes the Discrete Fourier Transform (DFT). (2) Information diffusion: GAT aggregates neighbor nodes with varying weights to via attention mechanisms. FourierGNN and GCN update node information via convoluting neighbor nodes. Different from GCN, FourierGNN assigns varying importance to neighbor nodes in different diffusion steps. We provide a detailed comparison in Appendix D.
4.3 Multivariate Time Series Forecasting with FourierGNN
In this section, we instantiate FourierGNN for MTS forecasting. The overall architecture of our model is illustrated in Figure 2. Given the MTS input data  $X_{t}\in \mathbb{R}^{N\times T}$ , first we construct a fully-connected hypervariate graph  $\mathcal{G}_t = (X_t^\mathcal{G},A_t^\mathcal{G})$  with  $X_{t}^{\mathcal{G}}\in \mathbb{R}^{NT\times 1}$  and  $A_{t}^{\mathcal{G}}\in \{1\}^{n\times n}$ . Then, we project  $X_{t}^{\mathcal{G}}$  into node embeddings  $\mathbf{X}_t^{\mathcal{G}}\in \mathbb{R}^{NT\times d}$  by assigning a  $d$ -dimension vector for each node using an embedding matrix  $E_{\phi}\in \mathbb{R}^{1\times d}$ , i.e.,  $\mathbf{X}_t^{\mathcal{G}} = X_t^{\mathcal{G}}\times E_\phi$ .