SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The interpretation of  $n$ -invariant FGO. An  $n$ -invariant FGO is similar to a shared-weight convolution kernel or filter of CNNs that slide along  $([n] \times [n])$  input features, which effectively reduces parameter volumes and saves computation costs. Note that although we adopt the same transformation (i.e., the  $n$ -invariant FGO) over  $NT$  frequency points, we embed the raw MTS inputs in the  $d$ -dimension distributive space beforehand and then perform FourierGNN over MTS embeddings, which can be analogized as  $d$  convolution kernels/filters in each convolutional layer in CNNs. This can ensure FourierGNN is able to learn informative features/patterns to improve its model capacity (See the following analysis of the effectiveness of  $n$ -invariant FGO).
The effectiveness of  $n$ -invariant FGO. In addition, the  $n$ -invariant parameterized FGO is empirically proven effective to improve model generalization and achieve superior forecasting performance (See the ablation study in Section 5.3 for more details). Although parameterizing  $\mathcal{F}(\kappa) \in \mathbb{C}^{n \times d \times d}$  (i.e., an  $n$ -variant FGO) may be more powerful and flexible than the  $n$ -invariant FGO in terms of forecasting performance, it introduces much more parameters and training time costs, especially in case of multi-layer FourierGNN, and may obtain inferior performance due to inadequate training or overfitting. As indicated in Table 6, the FourierGNN with the  $n$ -invariant FGO achieves slightly better performance than that with the  $n$ -variant FGO on ECG and COVID-19, respectively. Notably, the FourierGNN with the  $n$ -variant FGO introduces a much larger parameter volume proportional to  $n$  and requires significantly more training time. In contrast,  $n$ -invariant FGO is  $n$ -agnostic and lightweight, which is a more wise and efficient alternative. These results confirm our design and verify the effectiveness and applicability of  $n$ -invariant FGO.
Table 6: Comparison between FourierGNN models with  $n$ -invariant FGO and  $n$ -variant FGO on the ECG and COVID-19 datasets.
Datasets	Models	Parameters (M)	Training (s/epoch)	MAE	RMSE	MAPE (%)
ECG	n-invariant	0.18	12.45	0.052	0.078	10.97
n-variant	82.96	104.06	0.053	0.078	11.05
COVID-19	n-invariant	1.06	0.62	0.123	0.168	71.52
n-variant	130.99	7.46	0.129	0.174	72.12
C.3 Proof of Proposition 1 and Interpretation of FourierGNN
$$
\mathcal {F} ^ {- 1} (\mathcal {F} (X) \mathcal {S} _ {0: k}) = A _ {k: 0} X W _ {0: k}, \quad \mathcal {S} _ {0: k} = \prod_ {i = 0} ^ {k} \mathcal {S} _ {i}, A _ {k: 0} = \prod_ {i = k} ^ {0} A _ {i}, W _ {0: k} = \prod_ {i = 0} ^ {k} W _ {i}
$$
where  $A_0, S_0, W_0$  are the identity matrix,  $A_k \in \mathbb{R}^{n \times n}$  corresponds to the  $k$ -th diffusion step sharing the same sparsity pattern of  $A$ ,  $W_k \in \mathbb{R}^{d \times d}$  is the  $k$ -th weight matrix,  $\mathcal{S}_k \in \mathbb{C}^{d \times d}$  is the  $k$ -th FGO satisfying  $\mathcal{F}(A_k X W_k) = \mathcal{F}(X) \mathcal{S}_k$ , and  $\mathcal{F}$  and  $\mathcal{F}^{-1}$  denote DFT and its inverse, respectively.
Proof. The proof aims to demonstrate the equivalence between the recursive multiplication of FGOs in Fourier space and multi-order convolutions in the time domain. According to  $\mathcal{F}(A_kXW_k) = \mathcal{F}(X)\mathcal{S}_k$ , we expand the multi-order convolutions  $A_{0:K}XW_{0:K}$  in the time domain using a set of FGOs in Fourier space:
$$
\begin{array}{l} \mathcal {F} \left(A _ {K} A _ {K - 1} \dots A _ {0} X W _ {0} \dots W _ {K - 1} W _ {K}\right) = \mathcal {F} \left(A _ {K} \left(A _ {K - 1} \dots A _ {0} X W _ {0} \dots W _ {K - 1}\right) W _ {K}\right) \\ = \mathcal {F} \left(A _ {K - 1} \dots A _ {0} X W _ {0} \dots W _ {K - 1}\right) \mathcal {S} _ {K} \\ = \mathcal {F} \left(A _ {K - 1} \left(A _ {K - 2} \dots A _ {0} X W _ {0} \dots W _ {K - 2}\right) W _ {K - 1}\right) \mathcal {S} _ {K} \\ = \mathcal {F} \left(A _ {K - 2} \dots A _ {0} X W _ {0} \dots W _ {K - 2}\right) \mathcal {S} _ {K - 1} \mathcal {S} _ {K} \\ = \dots \\ = \mathcal {F} (X) \mathcal {S} _ {0} \dots \mathcal {S} _ {K - 1} \mathcal {S} _ {K} \\ = \mathcal {F} (X) \mathcal {S} _ {0: K} \tag {10} \\ \end{array}
$$
where it yields  $\mathcal{F}^{-1}(\mathcal{F}(X)\mathcal{S}_{0:K}) = A_{K:0}XW_{0:K}$  with  $\mathcal{S}_{0:K} = \prod_{i=0}^{K}\mathcal{S}_{i}, A_{K:0} = \prod_{i=K}^{0}A_{i}$  and  $W_{0:K} = \prod_{i=0}^{K}W_{i}$ . Proved.
Thus, the FourierGNN can be rewritten as (for convenience, we exclude the non-linear activation function  $\sigma$  and learnable bias parameters  $b$ ):
$$
\mathcal {F} ^ {- 1} \left(\sum_ {k = 0} ^ {K} \mathcal {F} (X) \mathcal {S} _ {0: K}\right) = A _ {0} X W _ {0} + A _ {1} \left(A _ {0} X W _ {0}\right) W _ {1} + \dots + A _ {K: 0} X W _ {0: K} \tag {11}
$$
From the right part of the above equation, we can observe that it assigns different weights to weigh the information of different neighbors in each diffusion order. This property enable FourierGNN to capture the complex correlations (i.e., spatiotemporal dependencies) in the hypervariate graph, which is empirically verified in our visualization experiments.
D Compared with Other Graph Neural Networks
Graph Convolutional Networks. Graph convolutional networks (GCNs) depend on the Laplacian eigenbasis to perform the multi-order graph convolutions over a given graph structure. Compared with GCNs, FourierGNN as an efficient alternative to multi-order graph convolutions has three main differences: 1) No eigendecompositions or similar costly matrix operations are required. FourierGNN transforms the input into Fourier domain by discrete Fourier transform (DFT) instead of graph Fourier transform (GFT); 2) Explicitly assigning various importance to nodes of the same neighborhood with different diffusion steps. FourierGNN adopts different Fourier Graph Operators  $S$  in different diffusion steps corresponding to different dependencies among nodes; 3) FourierGNN is invariant to the discretization  $N$ ,  $T$ . It parameterizes the graph convolution via Fourier Graph Operators which are invariant to the graph structure and graph scale.
Graph Attention Networks. Graph attention networks (GATs) are non-spectral attention-based graph neural networks. GATs use node representations to calculate the attention weights (i.e., edge weights) varying with different graph attention layers. Accordingly, both GATs and FourierGNN do
not depend on eigendecompositions and adopt varying edge weights with different diffusion steps (layers). However, FourierGNN can efficiently perform graph convolutions in Fourier space. For a complete graph, the time complexity of the attention calculation of  $K$  layers is proportional to  $Kn^2$  where  $n$  is the number of nodes, while a  $K$ -layer FourierGNN infers the graph structure in Fourier space with the time complexity proportional to  $n \log n$ . In addition, compared with GATs that implicitly achieve edge-varying weights with different layers, FourierGNN adopts different FGOs in different diffusion steps explicitly.
We use seven public multivariate benchmarks for multivariate time series forecasting and these benchmark datasets are summarized in Table 7.
Table 7: Summary of datasets.
Datasets	Solar	Wiki	Traffic	ECG	Electricity	COVID-19	METR-LA
Samples	3650	803	10560	5000	140211	335	34272
Variables	592	2000	963	140	370	55	207
Granularity	1hour	1day	1hour	-	15min	1day	5min
Start time	01/01/2006	01/07/2015	01/01/2015	-	01/01/2011	01/02/2020	01/03/2012
Solar<sup>2</sup>: This dataset is about solar power collected by National Renewable Energy Laboratory. We choose the power plant data points in Florida as the data set which contains 593 points. The data is collected from 2006/01/01 to 2016/12/31 with the sampling interval of every 1 hour.
Wiki [34]: This dataset contains a number of daily views of different Wikipedia articles and is collected from 2015/7/1 to 2016/12/31. It consists of approximately  $145k$  time series and we randomly choose  $2k$  from them as our experimental data set.
Traffic [34]: This dataset contains hourly traffic data from 963 San Francisco freeway car lanes. The traffic data are collected since 2015/01/01 with the sampling interval of every 1 hour.
$\mathbf{ECG}^3$ : This dataset is about Electrocardiogram(ECG) from the UCR time-series classification archive [38]. It contains 140 nodes and each node has a length of 5000.
Electricity<sup>4</sup>: This dataset contains the electricity consumption of 370 clients and is collected since 2011/01/01. The data sampling interval is every 15 minutes.
COVID-19<sup>5</sup>: This dataset is about COVID-19 hospitalization in the U.S. states of California (CA) from 01/02/2020 to 31/12/2020 provided by the Johns Hopkins University with the sampling interval of every one day.
METR-LA<sup>6</sup>: This dataset contains traffic information collected from loop detectors in the highway of Los Angeles County from 01/03/2012 to 30/06/2012. It contains 207 sensors and the data sampling interval is every 5 minutes.
E.2Baselines
In experiments, we conduct a comprehensive comparison of the forecasting performance between our FourierGNN and representative and state-of-the-art (SOTA) models as follows.
VAR [32]: VAR is a classic linear autoregressive model. We use the Statsmodels library (https://www.statsmodels.org) which is a Python package that provides statistical computations to realize the VAR.
2https://www.nrel.gov/grid/solar-power-data.html
<sup>3</sup>http://www.timeseriesclassification.com/description.php?Dataset=ECG5000
<sup>4</sup>https://archive.ics.uci.edu/ml/datasets/ElectricityLoadDiagrams20112014
<sup>5</sup>https://github.com/CSSEGISandData/COVID-19
<sup>6</sup>https://github.com/liyaguang/DCRNN
DeepGLO [34]: DeepGLO models the relationships among variables by matrix factorization and employs a temporal convolution neural network to introduce non-linear relationships. We download the source code from: https://github.com/rajatsen91/deepglo. We follow the recommended configuration as our experimental settings for wiki, electricity, and traffic datasets. For Covid datasets, the vertical and horizontal batch size is set to 64, the rank of the global model is set to 64, the number of channels is set to [32, 32, 32, 1], and the period is set to 7.
LSTNet [33]: LSTNet uses a CNN to capture inter-variable relationships and an RNN to discover long-term patterns. We download the source code from: https://github.com/laiguokun/LSTNet. In our experiment, we use the recommended configuration where the number of CNN hidden units is 100, the kernel size of the CNN layers is 4, the dropout is 0.2, the RNN hidden units is 100, the number of RNN hidden layers is 1, the learning rate is 0.001 and the optimizer is Adam.
TCN [8]: TCN is a causal convolution model for regression prediction. We download the source code from: https://github.com/locuslab/TCN. We utilize the same configuration as the polyphonic music task, exampled in the open source code where the dropout is 0.25, the kernel size is 5, the number of hidden units is 150, the number of levels is 4 and the optimizer is Adam.
Reformer [35]: Reformer combines the modeling capacity of a Transformer with an architecture that can be executed efficiently on long sequences and with small memory use. We download the source code from: https://github.com/thuml/Autoformer. We follow the recommended configuration as the experimental settings.
Informer [9]: Informer leverages an efficient self-attention mechanism to encode the dependencies among variables. We download the source code from: https://github.com/zhouhaoyi/Informer2020. We follow the recommended configuration as our experimental settings where the dropout is 0.05, the number of encoder layers is 2, the number of decoder layers is 1, the learning rate is 0.0001, and the optimizer is Adam.
Autoformer [4]: Autoformer proposes a decomposition architecture by embedding the series decomposition block as an inner operator, which can progressively aggregate the long-term trend part from intermediate prediction. We download the source code from: https://github.com/thuml/Autoformer. We follow the recommended configuration as our experimental settings with 2 encoder layers and 1 decoder layer.
FEDformer [27]: FEDformer proposes an attention mechanism with low-rank approximation in frequency and a mixture of expert decomposition to control the distribution shifting. We download the source code from: https://github.com/MAZiqing/FEDformer. We use FEB-f as the Frequency Enhanced Block and select the random mode with 64 as the experimental mode.
SFM [24]: On the basis of the LSTM model, SFM introduces a series of different frequency components in the cell states. We download the source code from: https://github.com/z331565360/State-Frequency-Memory-stock-prediction. We follow the recommended settings where the learning rate is 0.01, the frequency dimension is 10, the hidden dimension is 10 and the optimizer is RMSProp.
StemGNN [10]: StemGNN leverages GFT and DFT to capture dependencies among variables in the frequency domain. We download the source code from: https://github.com/microsoft/StemGNN. We use the recommended configuration of stemGNN as our experiment setting where the optimizer is RMSProp, the learning rate is 0.0001, the number of stacked layers is 5, and the dropout rate is 0.5.
MTGNN [13]: MTGNN proposes an effective method to exploit the inherent dependency relationships among multiple time series. We download the source code from: https://github.com/nnzhan/MTGNN. Because the experimental datasets have no static features, we set the parameter load(static feature to false. We construct the graph by the adaptive adjacency matrix and add the graph convolution layer. Regarding other parameters, we adopt the recommended settings.
GraphWaveNet [17]: GraphWaveNet introduces an adaptive dependency matrix learning to capture the hidden spatial dependency. We download the source code from: https://github.com/nnzhan/Graph-WaveNet. Since our datasets have no prior defined graph structures, we use only adaptive adjacent matrix. We add a graph convolution layer and randomly initialize the adjacent matrix. We adopt the recommended configuration as our experimental settings where the learning rate is 0.001, the dropout is 0.3, the number of epochs is 50, and the optimizer is Adam.
AGCRN [2]: AGCRN proposes a data-adaptive graph generation module for discovering spatial correlations from data. We download the source code from: https://github.com/LeiBAI/AGCRN. We follow the recommended configuration as our experimental settings where the embedding dimension is 10, the learning rate is 0.003, and the optimizer is Adam.
TAMP-S2GCNets [11]: TAMP-S2GCNets explores the utility of MP to enhance knowledge representation mechanisms within the time-aware DL paradigm. We download the source code from: https://www.dropbox.com/sh/n0ajd5l0tdeyb80/AABGn-ejfV1YtRwjf_L0A0sNa?dl=0. TAMP-S2GCNets requires predefined graph topology and we use the California State topology provided by the source code as input. We adopt the recommended configuration as our experimental settings on COVID-19.
DCRNN [18]: DCRNN uses bidirectional graph random walk to model spatial dependency and recurrent neural network to capture the temporal dynamics. We download the source code from: https://github.com/liyaguang/DCRNN. We follow the recommended configuration as our experimental settings with the batch size is 64, the learning rate is 0.01, the input dimension is 2 and the optimizer is Adam. DCRNN requires a pre-defined graph structure and we use the adjacency matrix as the pre-defined structure provided by the METR-LA dataset.
STGCN [1]: STGCN integrates graph convolution and gated temporal convolution through spatial-temporal convolutional blocks. We download the source code from:https://github.com/VeritasYin/STGCN_IJCAI-18. We use the recommended configuration as our experimental settings where the batch size is 50, the learning rate is 0.001 and the optimizer is Adam. STGCN requires a pre-defined graph structure and we leverage the adjacency matrix as the pre-defined structures provided by the METR-LA dataset.
CoST [36]: CoST separates the representation learning and downstream forecasting task and proposes a contrastive learning framework that learns disentangled season-trend representations for time series forecasting tasks. We download the source code from: https://github.com/salesforce/CoST. We set the representation dimension to 320, the learning rate to 0.001, and the batch size to 32. Inputs are min-max normalization, we perform a 70/20/10 train/validation/test split for the METR-LA dataset and 60/20/20 for the COVID-19 dataset.
E.3 Evaluation Metrics
We use MAE (Mean Absolute Error), RMSE (Root Mean Square Error), and MAPE (Mean Absolute Percentage Error) as the evaluation metrics in the experiments.
Specifically, given the groundtruth at timestamps  $t$ ,  $Y_{t} = [\pmb{x}_{t + 1}, \dots, \pmb{x}_{t + \tau}] \in \mathbb{R}^{N \times \tau}$ , and the predictions  $\hat{Y}_t = [\hat{\pmb{x}}_{t + 1}, \dots, \hat{\pmb{x}}_{t + \tau}] \in \mathbb{R}^{N \times \tau}$  for future  $\tau$  steps at timestamp  $t$ , the metrics are defined as follows:
$$
M A E = \frac {1}{\tau N} \sum_ {i = 1} ^ {N} \sum_ {j = 1} ^ {\tau} | x _ {i j} - \hat {x} _ {i j} | \tag {12}
$$
$$
R M S E = \sqrt {\frac {1}{\tau N} \sum_ {i = 1} ^ {N} \sum_ {j = 1} ^ {\tau} \left(x _ {i j} - \hat {x} _ {i j}\right) ^ {2}} \tag {13}
$$
$$
M A P E = \frac {1}{\tau N} \sum_ {i = 1} ^ {N} \sum_ {j = 1} ^ {\tau} \left| \frac {x _ {i j} - \hat {x} _ {i j}}{x _ {i j}} \right| \times 100 \% \tag{14}
$$
with  $x_{ij}\in Y_t$  and  $\hat{x}_{ij}\in \hat{Y}_t$
E.4 Experimental Settings
We summarize the implementation details of the proposed FourierGNN as follows. Note that the details of the baselines are introduced in their corresponding descriptions (see Section E.2).
Network details. The fully connected feed-forward network (FFN) consists of three linear transformations with LeakyReLU activations in between. The FFN is formulated as follows:
$$
\mathbf {X} _ {1} = \text {L e a k y R e L U} \left(\mathbf {Y} _ {t} ^ {\mathcal {G}} \mathbf {W} _ {1} + \mathbf {b} _ {1}\right)
$$
$$
\mathbf {X} _ {2} = \text {L e a k y R e L U} \left(\mathbf {X} _ {1} \mathbf {W} _ {2} + \mathbf {b} _ {2}\right) \tag {15}
$$
$$
\hat {Y} = \mathbf {X} _ {2} \mathbf {W} _ {3} + \mathbf {b} _ {3}
$$
where  $\mathbf{W}_1\in \mathbb{R}^{(Td)\times d_1^{fn}}$ ,  $\mathbf{W}_2\in \mathbb{R}^{d_1^{fn}\times d_2^{fn}}$  and  $\mathbf{W}_3\in \mathbb{R}^{d_2^{fn}\times \tau}$  are the weights of the three layers respectively, and  $\mathbf{b}_1\in \mathbb{R}^{d_1^{fn}}$ ,  $\mathbf{b}_2\in \mathbb{R}^{d_2^{fn}}$  and  $\mathbf{b}_3\in \mathbb{R}^\tau$  are the biases of the three layers respectively. Here,  $d_1^{fn}$  and  $d_2^{fn}$  are the dimensions of the three layers. In addition, we adopt a ReLU activation function in Equation 6.
Training details. We carefully tune the hyperparameters, including the embedding size, batch size,  $d_1^{fn}$  and  $d_2^{fn}$ , on the validation set and choose the settings with the best performance for FourierGNN on different datasets. Specifically, the embedding size and batch size are tuned over {32, 64, 128, 256, 512} and {2, 4, 8, 16, 32, 64, 128} respectively. For the COVID-19 dataset, the embedding size is 256, and the batch size is set to 4. For the Traffic, Solar, and Wiki datasets, the embedding size is 128, and the batch size is set to 2. For the METR-LA, ECG, and Electricity datasets, the embedding size is 128, and the batch size is set to 32.
To reduce the number of parameters, we adopt a linear transform to reshape the original time domain representation  $\mathbf{Y}_t^G\in \mathbb{R}^{NT\times d}$  to  $\mathbf{Y}_t\in \mathbb{R}^{N\times T\times d}$ , and map  $\mathbf{Y}_t$  to a low-dimensional tensor  $\mathbf{Y}_t\in \mathbb{R}^{N\times l\times d}$  with  $l < T$ . We then reshape  $\mathbf{Y}_t\in \mathbb{R}^{N\times (ld)}$  and feed it to FFN. We perform a grid search on the dimensions of FFN, i.e.,  $d_1^{fn}$  and  $d_2^{fn}$ , over  $\{32,64,128,256,512\}$  and tune the intermediate dimension  $l$  over  $\{2,4,6,8,12\}$ . The settings of the three hyperparameters over all datasets are shown in Table 8. By default, we set the diffusion step (layers)  $K = 3$  for all datasets.
Table 8: Dimension settings of FFN on different datasets. * denotes that we feed the original time domain representation to FFN without the dimension reduction.
Datasets	Solar	Wiki	Traffic	ECG	Electricity	COVID-19	META-LR
l	6	2	2	*	4	8	4
dfn1	64	64	64	64	64	256	64
dfn2	256	256	256	256	256	512	256
E.5 Details for Visualization Experiments
To verify the effectiveness of FourierGNN in learning the spatiotemporal dependencies on the hypervariate graph, we obtain the output of FourierGNN as the node representation, denoted as  $\mathbf{Y}_t^{\mathcal{G}} =$  IDFT(FourierGNN(Xt))  $\in \mathbb{R}^{NT\times d}$  with Inverse Discrete Fourier Transform (IDFT). Then, we visualize the adjacency matrix  $\mathbf{A}$  calculated based the flatten node representation  $\mathbf{Y}_t^{\mathcal{G}}\in \mathbb{R}^{NT\times d}$  formulated as  $\mathbf{A} = \mathbf{Y}_t^{\mathcal{G}}(\mathbf{Y}_t^{\mathcal{G}})^T\in \mathbb{R}^{NT\times NT}$ , to show the variable correlations. Note that  $\mathbf{A}$  is normalized via  $\mathbf{A} / \max (\mathbf{A})$ . Since it is not feasible to directly visualize the huge adjacency matrix  $\mathbf{A}$  of the hypervariate graph, we visualize its different subgraphs in Figures 3, 4, 9, and 10 to better verify the learned spatiotemporal information on the hypervariate graph from different perspectives.
Figure 3. We select 8 counties and visualize the correlations between 12 consecutive time steps for each selected county respectively. Figure 3 reflects the temporal correlations within each variable.
Figure 4: On the METR-LA dataset, we average its adjacency matrix  $\mathbf{A}$  over the temporal dimension (i.e., marginalizing  $T$ ) to  $\mathbf{A}' \in \mathbb{R}^{N \times N}$ . Then, we randomly select 20 detectors out of all  $N = 207$  detectors and obtain their corresponding sub adjacency matrix  $(\mathbb{R}^{20 \times 20})$  from  $\mathbf{A}'$  for visualization. We further compare the sub-adjacency with the real road map (generated by the Google map tool) to verify the learned dependencies between different detectors.
Figure 9. Since we adopt a 3-layer FourierGNN, we can calculate four adjacency matrices based on the spectrum input  $\mathcal{X}_t^{\mathcal{G}}$  of FourierGNN and the outputs of each layer in FourierGNN. Following the
way of visualization in Figure 4, we select 10 counties and two timestamps on the four adjacency matrices for visualization. Figure 9 shows the effects of each layer of FourierGNN in filtering or enhancing variable correlations.
Figure 10. On the COVID-19 dataset, we randomly select 10 counties out of  $N = 55$  counties and obtain their four sub-adjacency matrices of four consecutive days for visualization. Each of the four sub adjacency matrices  $\mathbb{R}^{10 \times 10}$  embodies the dependencies between counties in one day. Figure 10 reflects the time-varying dependencies between counties (i.e., variables).