Edge-Varying Fourier Graph Network for Multivariate Time Series Forecasting
Authors: Kun Yi
Qi Zhang
Venue: Submitted to ICLR 2023 (2023)
Keywords: fourier graph convolution
fourier graph shift operator
supra-graph
multivariate time series forecasting

Abstract: The key problem in multivariate time series (MTS) analysis and forecasting aims to disclose the underlying couplings between variables that drive the co-movements. Considerable recent successful MTS methods are built with graph neural networks (GNNs) due to their essential capacity for relational modeling. However, previous work often used a static graph structure of time-series variables for modeling MTS failing to capture their ever-changing correlations over time. To this end, a fully-connected supra-graph connecting any two variables at any two timestamps is adaptively learned to capture the high-resolution variable dependencies via an efficient graph convolutional network. Specifically, we construct the Edge-Varying Fourier Graph Networks (EV-FGN) equipped with Fourier Graph Shift Operator (FGSO) which efficiently performs graph convolution in the frequency domain. As a result, a high-efficiency scale-free parameter learning scheme is derived for MTS analysis and forecasting according to the convolution theorem. Extensive experiments show that EV-FGN outperforms state-of-the-art methods on seven real-world MTS datasets.

Under review as a conference paper at ICLR 2023
FOURIER GRAPH NETWORKS FOR MULTIVARIATE TIME SERIES FORECASTING
Anonymous authors Paper under double-blind review
ABSTRACT
The key to multivariate time series (MTS) analysis and forecasting is to disclose the underlying couplings between variables that drive the co-movements. Con- siderable recent successful MTS methods are built with graph neural networks (GNNs) due to their essential capacity for relational modeling. However, previ- ous work often used static graph structure of time-series variables for modeling MTS, but failed to capture their ever-changing correlations over time. this paper, we build fully-connected supra-graphe representing non-static correla- tions between any two variables at any two timestamps, to capture high-resolution spatial-temporal dependencies. Whereas, conducting graph convolutions supra-graph heavily increases computational complexity. As a result, we propose the novel Edge- Margen Fourier Graph Networks (EV-FGN), which reformulates the graph convolutions in the frequency domain with high efficiency and scale-free parameters, and applies edge varying graph filters capture the time-varying variable dependencies. Extensive experiments show that EV-FGN outperforms tfate-ofthhe- art methods on seven real-world MTS datasets.
such a
INTRODUCTION
Multivariate time series (MTS) forecasting is key ingredient in many real-world scenarios, including
weather forecasting Zheng et al. (2015). decision making Borovykh et al. (2117). traffic forecasting
Yu et al. (2) Bai et al. (2020). COVID-19 prediction Cao et al. (2020); Chen et al. (2022). etc.
Recently, deep neural networks, such as long short-term memory (LSTM) Hochreiter & Schmid.
huber (1997), convolutional neural network (CNN) Borovykh et al. (017). Transformer Vaswani et al. (2017), have dominated MTS modeling. In particular, graph neural networks (GNNs) have demonstrated promising performance on MTS forecasting with their essential capability to capture the complex couplings between time-series variables. Some studies enable adaptively learn the graph for MTS forecasting even without an explicit graph structure, e.g., by node similarity Mateos let al. (2019); Bai et al. (2020); Wu et al. (2019) and/or self-attention mechanism Cao al. 2020).
Despite the success of GNNs on MTS forecasting, three practical challenges are eagerly demanded to address: 1) the dependencies between each pair of time series variables generally non-static, which demands full dependencies modeling with all possible lags; 2) a high-efficiency dense graph learning method is demanded replace the high-cost operators of graph convolutions and attention (with quadratic time complexity of the graph size); 3) the graph over MTS varies with different temporal interactions, which demands an efficient dynamic structure encoding method. In this paper, different from most GNN-based methods that construct graphs to model the spatial correlation between variables (i.e. nodes) Bai et al. (2020); ; Wu al. (2019) 2020). we attempt to build supra-graph that sheds light on the "high-resolution correlations between any two variables at any two timestamps (i.e., fine- aee-ggained spatial-tempont dependencies), and largely enhances the expressiveness non-static spatial-temporal dependencies.
Obviously, the supra-graph will heavily increase the computational complexity of GNN-based model, then high-efficiency learning method is required to reduce the cost for model training. Inspired by Fourier Neural Operator (FNO) Li et al. (202I), ). we reformulate the graph convolution (time domain) to much lower-complexity matrix multiplication in the frequency domain by leveraging efficient and aewly-defined Fourier Graph Shift Operator (FGSO). In addition, it is necessary to consider the multiple iterations (layers) of graph convolutions expand receptive neighbors and mix the diffusion

---

Under review as a conference paper at ICLR 2023
information in the graph. To capture time- variing diffusion over the supra-graie graah, we introduce the edge-varying graph filter to weight the graph edges differently from different iterations and reformulate the edge-varying graph filter with multiple frequency-invare invarrint FGSOs in the frequency domain to reduce the computational cost of graph convolutions. Finally, a novel Edge-Vare Varying Fourier Graph Networks (EV-FGN) is designed for MTS analysis, which is stacked with multiple FGSOs to perform high-efficient multi-layer graph convolutions in the Fourier space.
The main contributions of this paper are summarized follows:
We adaptively learn a supra-graph, representing non-static correlations between any two variables any two timestamps, to capture high-resolution spatial-temporal dependencies.
To efficiently compute graph convolutions over the supra-graph, we reformulate the graph convolu- tions in the Fourier space by leveraging FGSO. To the best of knowledge, this work makes the first step to reformulate the graph convolutions in the Fourier space.
We design novel network EV-FGN evolved from edge-varying graph filters for MTS analysis to capture the time-varying variable dependencies in the Fourier space. This study makes the first attempt to design a complex-valued ford-forward network in the Fourier space to efficiently compute multi-layer graph convolutions.
Extensive experimental results on seven MTS datasets demonstrate that EV-FGN achieves state-of- the-art performance with high efficiency and fewer parameters. Multifaceted visualizations further interpret the efficacy of EV-FGN in graph representation learning for MTS forecasting.
2
RELATED WORKS
2.1 MULTIVARIATE TIME SERIES FORECASTING
Classic time series forecasting methods are linear models, such VAR Watson (993), ARIMA
Asteriou & Hall (201) 1 and state space model (SSM) Hyndman et al. 2008 Recently, deep learning based methods Lai et al. (2018)][Sen et al. (2019); Zhou et al. (2021) have dominated MTS forecasting
due to their capability of fitting any complex nonlinear correlations Lim & Zohren 222TT)
MTS with GNN. More recently, MTS have embraced GNN Wu et al. (2019); Bai et al. (2020) Wu et al. (2020); Yu et al.] (b018b) Chen et al.) (2022); Li et al. (2018) due to their best capability of modeling structural dependencies between variables. Most of these models, such as STGCN Yu et al.) (2018b), DCRNN Li et al. (018) and TAMP-S2GCNets Chen al. (2022), require pre-defined graph structure which is usually unknown in most cases. In recent years, some GNN-based works Kipf et al. 2018); Deng & Hooi (2021) account for the dynamic dependencies due to network design such as the Lume-arrying attention Deng & Hooi (00TT). In comparison, our proposed model captures the dynamic dependencies leveraging the high-resoluttin correlation in the supra-graph without introducing specific networks.
MTS with Fourier transform. Recently, increasing MTS forecasting models have introduced the Fourier theory into neural networks as high-efficiency convolution operators Guibas et al. (2022) ;
Chi al. (020)). SFM Zhhnn et al. (2017) decomposes the hidden state of LSTM into multiple Trequencies by discrete Fourier transform (DFT). mWDN [Wang et al. (2018) decomposes the time series into multilevel sub-series by discrete wavelet decomposition and feeds them LSTM network, respectively. ATFN Yang et al. (2022) utilizes time-domain block to learn the trending feature of complicated non-stationary time series and rrequency-domal block to capture dynamic and complicated periodic patterns of time series data. FEDformer Zhou et (2022) proposes an attention mechanism with low-rank approximation in frequency and a mixture of expert decomposition to control the distribution shift. However, these models only capture temporal dependencies in the frequency domain. StemGNN Cao et al. 2020) takes the advantages of both inter-series correlations and temporal dependencies by modeling them in the spectral domain, but, it captures the temporal and spatial dependencies separately. Unlike these efforts, our model is able to jointly encode spatial-temporal dependencies in the Fourier space.
2

---

Under review as a conference paper at ICLR 2023
Edge Varying Fourler Graph Network
Embedding
20 Hra
Input
Inpur
Output
FGSO
FGSO
FGSO
X
Xxp
X
3,5,2
Time Space
Fourier Space
Time Space
Figure 1: The network architecture of our proposed model. Given an input X E RNXT we I) embed into X E Nxx'xx 2) transform to Fourier space E CNxX'x by 2D DFT on the discrete N x T spatial-temporal space; 3) perform graph convolutions in Fourier space by conducting multiplication of FGSO S and X' in the K K-layer EV-FGN; transform the output of EV-FGN to time domain X E T xTxxd by 2D IDFT; 5) generate r-step predictions X RNN via feeding X to two-layer feed- ferd-forward network.
2.2 GRAPII SHIFT OPERATOR
Graph shift operators (GSOs) (e.g.. the adjacency matrix and the Laplacian matrix) are a general set of linear operators which are used to encode neighbourhood topologies in the graph. Kliepera let al. (2019) shows that applying the varying GSOs in the message passing step of GNNs can lead significant improvement of performance. Dasoulas al. (2021) proposes parameterized graph shift operator to automatically adapt to networks with varying sparsity. Isufi et al. (202 allows different nodes to use different GSOs to weight the information of different neighbors, Hadou et al. (2022) introduces a linear composition of the graph shift operator and time-shift operator to design space-time filters for vime-varying graph signals. Inspired by these works, in this paper we design varying parameterized graph shift operator in Fourier space.
2.3 FOURIER NEURAL OPERATOR
Different from classical neural networks which learn mappings between spaces, neural operators learn mappings between (2021b). Fourier neural operators (FNOs), currently the most promising one of the neural operators, are universal, in the sense that they can approximate any continuous operator to the desired accuracy Kovachki et al. (2021a). Li et al. (202I) formulates new neural operator by parameterizing the integral kernel directly in the Fourier space, allowing for expressive and efficient architecture for partial differential equations. Guibas et al. (2022) proposes an efficient token mixer that learns to mix in the Fourier domain which is principled architectural modification FNO. In this paper, learn a Fourier graph shift operator by leveraging the Fourier Neural operator.
Euclidean function spaces Kovachki et al.
METHODOLOGY
Let us denote the entire MTS raw data as X E NXL with N variables and timestamps. Under the rolling setting, we have window-sizer time-series inputs with T timestamps, i.e., {X X C x, e NxT). Accordingly, we formulate the problem of MTS forecasting as learning the spatial- temporal dependencies simultaneously supra-graph G contains N nodes that represent values of each variable at each timestamp in graph shift operator (GSO) representing the connection structure of G. Since the underlying graph is unknown in most MTS scenarios, assume all nodes in the supra-graph are connected with each other, i.e., a fully-connected graph, and perform graph convolutions on the graph to learn spatial-temporal representation. Then, given the observed values of previous? T steps at timestamp
E
supra-graph = (X, attributed to each X. The
X, and S E (N+T)
f, i.e., Xt-Txt E R NxT, the task of multi-step multivariate time series forecasting is to predict the values of N variables for next steps denoted as f++114 E <RNx on the supra-graph G, formulated follows:
H++H+ F(Xt-TT 9:
(1)
where F is the forecasting model with parameters e.
3

---

Under review as a conference paper at ICLR 2023
3.1 OVERALL ARCHITECTURE
The overall architecture of our model is illustrated in Fig. Given input data X E RNXI, first we embed the data into embeddings X E RRXxxx by assigning d-dimension vector for each node using an embedding matrix $ E RNxTxd, i.e. X == X x t. Instead of directly learning the huge embedding matrix, we introduce two small parameter matrices; 1) a variable embedding matrix 6, E IR RR xxxxxx and 2) a temporal embedding matrix O₂ E R RlxTxd to factorize 4, i.c., & = Ov X Φu. Subsequently, we perform 2D discrete Fourier transform (DFT) on each discrete N 7 spatial- temporal plane of the embeddings X and obtain the frequency input X' == DFT(X) E CNxTxd We then feed X to K-layer Varying Fourier Graph Networks (denoted as V k) to perform graph convolutions for capturing the spatial-temporal dependencies simultaneously in the Fourier space.
To make predictions time domain, we perform 2D inverse Fourier transform to generate repre- sentations X == IDFT(W v x (X)) R in the time domain. The representation is then fed to two-layer weed-forward networks (FFN, see more details in Appendix [.4) parameterized with weights and biases denoted as Φf to make predictions for future T steps X E NRN by one forward procedure. The L2 loss function for multi-step forecasting can be formulated as:
x(X:x;0)- - ||| - -x
with parameters O (ov Du Offi V x and the groudtruth X++1:t+7 C X at timestamp f.
3.2 FOURIER GRAPH SHIFT OPERATOR
According to the discrete signal processing on graphs Sandryhaila & Moura (2013), graph shift operator (GSO) is defined general family of operators which enables the diffusion of information over graph structures Gama et al. (2020); Dasoulas et al.] (2021).
Definition (Graph Shift Operator). Given graph G with nodes, matrix S E g= is called a Graph Shift Operator (GSO) if it satisfies S.j = 0 if + j and nodes i, are not connected.
The graph shift operator includes the adjacency, Laplacian matrices and their normalisations as instances of its class, and represents the connection structure of the graph. Accordingly, given the graph G' attributed to X E Raxd (corresponding to the supra-graph with N T'), general form of spatial- based graph convolution is defined as
O(X) == SXW
with the parameter matrix W E Raxd, Regarding S kernel : [r]] x [n] + idxdd with k[i,j] = S.y W, where [n] = {1. 2, .n.. Then the graph convolution can be viewed as kernel summation.
scores, define matrix-
O(X)
Σ
[i
Vi E [n].
In the special case of the Green's 's kernel k[V,,]] []] k(i j]. we can rewrite the kernel summation
O(x) Xbiliii - = (X * x)
Vi Vi
[n]. [n].
(5)
J=1
with X x denotes the convolution of discrete sequences X and k. According to the convolution
theorem Katznelson (1970) (see Appendix B). the graph convolution is rewritten as
O(x)()== F- (F(X) (i) Vi (n).
where F and F-1 denote the discrete Fourier transform (DFT) and its inverse (IDFT), respectively. The multiplication in the Fourier space a lower-complexity computation compared to the graph convolution, and DFT can be efficiently implemented by the fast Fourier transform (FFT).
Definition 2 (Fourier Graph Shift Operator). Given graph G (X,S) with input e Raad and GSO S e x and the weight matrix W E pdddd, the graph convolution is formulated
F(SXW) = F(X) Xn F(R)
where F denotes DFT, satisfies k[i.j] kli - j). and X is matrix multiplication on dimensions except that of n. We define S == F(x) Coxd Fourier graph shift operator (FGSO).
4

---

Under review as a conference paper at ICLR 2023
In particular, turning to case of the cunnecco supra-graph 9 with an all-one GSO S [1jaxn, it yields the space-invarian kernel j|i = Sty o W = W and F(SXW) = F(k) Accordingly, we can parameterize FGSO S with a complex-vali valledd matrix Cdxd which is frequency- invariant and is computationally low costly compared to varying kernel resulting parameterized to 2D discrete space, i.e., from [m] to [N] x [T), corresponding to the finite discrete spatial-temporal space of multivariate time series. See Appendix C for more explanations on the ieqquency-invariann FGSO and the extension to 2D domain.
matrix of Coxdxd Furthermore, we can extend Definition 2
Remarks. Compared with FGSO and FNO, frequency-invariant FGSO has several advantages. Assume a graph with n nodes and the embedding dimension d (d S nt). 1) Efficiency: the time complexity of frequency-invariant FGSO is ( log n + nd²) for DFT, IDFT and the matrix multiplication compared with that of O(n'd+ + nd² on GSO. 2) Scale-free parameters: frequency- invariant FGSO strategically shares (dd) parameters for each node and the parameter volume is agnostic to the data scale, while the parameter count of FNO is Ond²).
3.3 EDGE- VARYING FOURIER GRAPH NETWORKS
Graph filters as core operations in signal processing are linear transformations expressed as polyno- mials of the graph shift operator Isufi et al. (202T); Mateos et al. (2019) and can be used to exactly model graph convolutions and capture multi-order diffusion on graph structures Segarra et al. (2017). To capture the time-varying counterparts and adopt different weights to weight the information of different neighbors in each diffusion order, the edge-variant graph filters are defined as follows Isufi al. 2021); b Segarra al. (017)): given GSO S E TR=x corresponding to graph with n nodes,
K
HEV = So S + + SK = Σ SkO0
(8)
k=
where Sn denotes the identity matrix, {S ) is collection of K edge-weighting GSOs sharing the sparsity pattern of S. and S₃ E R corresponds to the k-th diffusion step. The edge-variant graph filters proved effective yield highly discriminative model and lay the foundation for the unification of GCNs and GAT's |sufi et al. (202T). We can extend Equation to HE and reformulate the multi-order graph convolution in recursive form, where we omit the weight W for conciseness.
Proposition 1. Given a graph input X E Rnnd the K K-order graph convolution under the edge- variant graph filter HEV En= Swo with {Sa= E BRxx1K Jk= is reformulated follows:
K F(X) x Souk s.l. So.k So Xn ... S.. X SK
HEVX= F-1
km
where S₁. @nxdxd is the k-th FGSO satisfying F(S,X)= = F(X) x, SR. So is the identity matrix, and F and F-1 denote the discrete Fourier transform and its reverse, respectively.
Proposition 1 proved in Appendix [.I states that corresponding to HEV as a summation of a recursive multiplication of individual FGSO in the Fourier space. Corresponding to our case of the supra-graph G, we can similarly adopt K frequency-invario FGSOs and parameterize each FGSO S₁ with complex-valuee matrix Cd This saves large amount of computation costs and results a concise form:
rewrite the multi-order graph convolution
X Σ F(X)Sook
8.1. So.k
II S ==
(10)
The recursive composition has a nice property that So.k = So.k 1 which inspires us to design a complex-valued feed forward network with Sş being the complex weights for the k-th layer. However, both the FGSO and edge-variant graph filter are linear transformations, limiting the capability of modeling non-linear information diffusion on graphs. Following by the convention in GNNs, we introduce the non- non-linear activation and biases to reformulate the k-the layer as follows:
X ( bk))
(11)
with the complex weight S, E Odxd, biases by E Cd and the activation function a. Accordingly, we design an edge-varying Fourier graph network (EV-FGN) in the Fourier space according to Equations
5

---

Under review as a conference paper at ICLR 2023
9 and 11
shown in Fig. Accordingly, the K-layer EV-FGN Vx is formulated:
K
M MK(N)
Σ
Xs s.t. X = bn)
(12)
k=0
where Xo == F(X), (S,- E [ and (hg E C complex-vale valuue parameters. The frequency output (X') then transformed via IDFT, followed with feed forward networks (see Appendix F.4 for more details) to make multi-step forecasting in the time domain.
Remarks. EV-FGN efficiently learns edge- and reighbor-dependeer weights to compute multi-order fly in the Fourier space with scale-free parameter volume. For K iterations of graph convolutions, GCNs have a general time complexity of OK''d + Knd)), and FNO needs O(Knd log n + Knd²), and EV-FGN achieves time complexity of (nd Kndd K'dd). EV-FGN saves time costs from FNO Fourier transforms and is much more efficient than GCNs, especially in our case with NT for given raw input X E RNT, See Appendix E for more details about the relations and differences between EV-FGN with FNO, adaptive FNO, and GNNs.
graph convolutions
4 EXPERIMENTS
4.1 SETUP
Datasets. We select seven representative datasets from different application scenarios for evalua- tion, including traffic, energy, web traffic, electrocardiogram, and COVID-19, These datasets are summarized Table All datasets are normalized using the min-max normalization. Except the COVID-19 dataset, we split the other datasets into training, validation, and test sets with the ratio of 7:2:1 in chronological order. For the COVID-19 dataset, the ratio is 6:2:2.
Table 1: Summary of datasets.
Datasets
Solar
Wiki
Traffic
ECG
Electricity
COVID-19
METR-LA
Samples
3650
803
10560
5000
140211
335
34272
Variables
592
2000
963
140
370
55
207
Granularity
hhour
Iday
1hour
15min
Iday
Smin
Start time
01/01/2006
01/07/2015
01/01/2015
01/01/2011
01/02/2020
01/03/2012
Baselines. We compare the forecasting performance of our EV-FGN with other representative and SOTA models the seven datasets, including VAR Watson (193)), SFM SFM Zhang et al. (2017) LSTNet
Lai ct al. (2118). TCN Bai et al. (2018), GraphWaveNN Wu et al. CiemGNNCCao et al. 2020)) MTGNN Wu et al. (2020), AGCRN Bai et al. (2020), Reformer Kitaev et al. (2020), Informer Zhou et al. (2021). Autoformer Wu et al. (2021). I) FEDformer Zhou et al.
2019). DeepGLO Sen et al. (201))
(2022) and Woo al. (2022) In addition, compare EV-FGN with SOTA TAMP-S2GCNets
Chen et al. (2022), DCRNN Li et al. (0018) and STGCN Yu et al. (2018a). which need pre-defined
graph structures.
Experimental Settings. All experiments are implemented in Python in Pytorch 1.8 (SFM in Keras) and conducted on one NVIDIA RTX 3080 card. Our model is trained using RMSProp with learning rate of 0.00001 and MSELoss (Mean Squared Error) as the loss function. The best parameters for all comparative models are chosen through careful parameter tuning on the validation set. We use MAE (Mean Absolute Errors), RMSE (Root Mean Squared Errors), and MAPE (Mean Absolute Percentage Error) to measure the performance.
More details about the datasets, baselines and experimental settings can be found in Appendix
4.2 RESULTS
We summarize the evaluation results in Table and more results can be found in Appendix G Generally, model EV-FGN establishes new of-t-oofthhearr all datasets. On average, EV-FGN improves 8.8% on MAE and 10.5% on RMSE compared to the best baseline for all datasets. Among these baselines, Reformer, Informer, Autoformer and FEDformer are transformer-bast models

---

Under review as a conference paper at ICLR 2023
that achieve competitive performance on Electricity and COVID-19 datasets since those models are competent in capturing temporal dependencies. However, they are defective in capturing the spatial dependencies explicitly. GraphWaveNett. MTGNN, StemGNN and AGCRN are GNN-based models that show promising performances on Wiki, Traffic, Solar and ECG datasets due to their high capability in handling spatial dependencies among variables. However, they limited to simultancously capturing spatial-temports dependencies. EV-FGN significantly outperforms the baseline models since it learns comprehensive spatial-temporal dependencies simultancously and attends to time-varying dependencies among variables. In Appendix G we further report the results on four datasets and show the comparison between EV-FGN with those models requiring pre-defined graph structures.
Table 2: Forecasting results on the six datasets.
Models
Solar
Wiki
Traffic
MAE
RMSE
MAPE(%)
MAE
RMSE
MAPE($)
MAE
RMSE
MAPE(%)
VAR Watson 1993
0,184
0.234
577.10
0.057
0.094
96.58
0.535
1.133
550.12
SFM Zhang
2017
0.161
0.283
362.89
0.081
0.156
104.47
0.029
0.044
59.33
LSTNe li et al. 2018
0.148
0.200
132.95
0.054
0.090
118. 24
0.026
0.057
25.77
TCN Bai
2018
0.176
0.222
142.23
0.094
0.142
99.66
0.052
0.067
Deep Ses u.. 2019
0.178
0.400
346.78
0.110
0.113
119.60
0.025
0.037
33.32
Reformet
cl 20,20
0,234
0.292
128.58
0.048
0.085
73.61
0.029
112.58
Informer Thou ci al.
702T
0,151
0.199
128.45
0.051
0.086
80.50
0.020
0.033
59.34
Aurofommr Wi et
20z
0,150
0.193
103.79
0.069
0.103
121.90
0.029
0.043
100.02
FEDfomer Zhou et cl.
2022
0.139
0.182
100.92
0.068
0.098
123.10
0.025
0.038
85.12
Graph:aveeee et et 2019
0,183
0.238
603
0.061
0.105
136.12
0.013
0.034
33,78
StemGNN Cao et
207211
0.176
0.222
128.39
0.190
0.255
117.92
0.080
0.135
64.51
MTGNN Wi et al.
2020
0,151
0,207
507.91
0.101
0.140
122. 96
0.013
0.030
29.53
AGCRN
2020
0.123
0.214
353.03
0.044
0.079
78.52
0.084
0.166
31.73
EV-FGN(ours)
0.120
0.162
116.48
0.041
0.076
64.50
0,011
0.023
28.71
Impryvment
2.4%
11.0%
6.8%
3.8%
12.4%
15.4%
23.3%
Models
ECG
Electricity RMSE
COVID-19
MAE
RMSE
MAPE(%)
MAE
MAPE(3)
MAE
MAPE(S)
VAR Watson
9933
0.120
0.170
22.56
0.101
0.163
43.11
0.226
0.326
191.95
SFM Zhang Ct al 2017
0.095
0.135
24.20
0.086
0.129
33.71
0.205
0.308
76.08
LSTNe
0.079
0.115
18.68
0.075
0.138
29.95
0.248
0.305
89.04
TCN
Bai cl ml 2018
0.078
0.107
17.59
0.057
0.083
26.64
0.317
0.354
151.78
Deep 00 Nen
2019
0.163
43.90
0.090
0.131
29.40
0.169
75.19
Reformer Kitacy et
2020
0.062
0.090
13.58
0.078
0.129
33.37
0.152
0.209
132.78
Informer Zhoo c.
20:4
0,056
0.085
11.99
0.070
0.119
32.66
0,200
0.259
155.55
Autofomee Wu
20
0.055
0.081
11.37
0.056
0.083
25.94
0.159
0.211
136,24
FEDformer Zhou ei ml.
2
0.055
0.080
11.16
0.055
0.081
25.84
0.160
0.219
134.45
Graph WavehWW
22199
0.093
0.142
40.19
0.094
0,140
37.01
0.201
0.255
100.83
StemGNN Cao al 2020
0,100
0.130
29.62
0.070
0.101
0.421
0.508
141.01
MIGNN Wu
0.090
0.139
35.04
0.077
0.113
29.77
0.394
88.13
AGCRN Har al. 2110
0.055
0.080
11.75
0.074
0,116
26.08
0.254
0.309
58,58
EV-FGN(ours)
0.052
0.078
11.05
0.051
0.077
24.28
0.129
0.173
71.52
Imprvement
5.5%
2.5%
1.0%
7.3%
4.9%
6.0%
15.1%
17.2%
4.3 ANALYSIS
Efficiency Analysis. We investigate the parameter volumes and training time costs of EV-FGN, StemGNN, AGCRN, GraphWaved and MTGNN on two representative datasets (Wiki and Traffic). We report the parameter volumes and the average time costs of five rounds of experiments in Table 3 In terms of parameter volumes, EV-FGN requires the least volume of parameters among the compara- tive models. Specifically, it achieves 32.2% and 9.5% parameter reduction over Graph WaveNet on Traffic and Wiki datasets, respectively. This is highly attributed that EV-FGN has shared scale-free parameters for each node. Regarding the training time, EV-FGN runs much faster than all baseline models, and shows 5.8% and 23.3% efficiency improvements over the fast baseline Graph WaveNet on Traffic and Wiki datasets, respectively. Besides, the variable count of Wiki dataset is about twice larger than that of Traffic dataset, but EV-FGN shows larger efficiency gaps with the basclines. These results demonstrate that EV-FGN shows high efficiency in computing graph convolutions and is scalable to large datasets with large graphs. Significantly, the supra- supra-graph in EV-FGN has N x T nodes, which is much larger than the graphs (with N nodes) in the baselines.
Alltion study. We conduct the ablation study on METR-LA dataset to evaluate the contribution of different components of our model. The results shown in Table 4 verify the effectiveness of
7

---

Under review as a conference paper at ICLR 2023
Table 3: Results of parameter volumes and training time costs Traffic and Wiki datasets.
Traffic
Wiki
models
Parameters
Training (s/epoch)
Parameters
Training (slepoch)
StemGNN 1606140
185.86+-2.222
4102406
02.95-+1.339
MTGNN
707516
169.34 1.56
1533436
28.69-+0.83
AGCRN
749940
113,46=1.91
755740
22.48=t1.01
GraphWaveNet
280860
105.3811.24
292460
21.23:10.76
EV-FGN (ours)
190564
99.25:11.07
264804
16.28:10:48
each component. Specifically, w/o Embedding shows the significance of performing embedding to improve model generalization. w/o Dynamic Filter using the same graph shift operator verifies the effectiveness of applying different graph shift operators in capturing time-varye varying dependencies. In addition, w/o Residual represents EV-FGN without the K layer, while w/o Summation adopts the last order (layer) output Xx as the output of EV-FGN. These results demonstrate the importance of high-order diffusion and the contribution of multi-order diffusion. More results and analysis of the ablation study provided in Appendix H.
Table 4: Ablation study the METR-LA dataset.
metrics
w/o Embedding
w/o Dynamic Filter
w/o Residual
w/o Summation
EV-FGN
MAE
0.053
0.055
0.054
0.054
0.050
RMSE
0.116
0.114
0.115
0.114
0.113
MAPE(%)
86.73
86.69
86.75
86.62
86.30
4.4
VISUALIZATION
To better investigate the spatial-temporal representation learned by EV-FGN, conduct more visualization experiments on METR-LA and COVID-19 datasets. More details about the way of visualization be found in Appendix R5
Glendalee
pmoodd
Angeles
Figure 2: The adjacency matrix (right) learned by EV-FGN and the corresponding road map (left).
Visualization spatial representations learned by EV-FGN. We produce the visualization of the generated adjacency matrix according to the learned representation from EV-FGN on the METR-LA dataset. Specifically, we randomly select 20 detectors and visualize their corresponding adjacency matrix via heat map, shown in Fig. 2 Correlating the adjacency matrix with the real road map, we observe: 1) the detectors (7, 8, 9. 11, 13, 18, and 19) are very close in physical distance, corresponding to the high values of their correlations with each other in the heat map; 2) the detectors 4, 14 and 16 have small overall correlation values since they are far from other detectors; 3) however, compared with detectors 14 and 16, the detector 4 has slightly higher correlation values other detectors e.g.,
8

---

Under review as a conference paper at ICLR 2023
7, 8, 9, which is attributed that although they are far apart, the detectors 4. 7, 8, 9 are the same road. The results verify the effectiveness of EV-FGN learning highly interpretative correlations.
Visualization four consecutive days. Furthermore, adjacency matrix of randomly-sclected counties on four consecutive days on the COVID- 19 dataset. The visualization results via heat map shown in Fig. 3 From the figure,
conduct experiments to visualize the
observe that EV-FGN learns clear spatial patterns that show continuous evolution in the time dimension. This is because EV-FGN highlights the edge-varying design and attends to the time-varying variability of the supra-graph. These results verify that our model enjoys the feasibility of exploiting the time-varying dependencies among variables.
Figure 3: The adjacency matrix for four consecutive days the COVID-19 dataset.
Visualization of EV-FGN diffusion process. To understand how EV-FGN works, we analyze the frequency input of each layer. We choose 10 counties from COVID-19 dataset and visualize their adjacency matrices at two different timestamps, as shown in Fig. From left to right, the results correspond Xn, X3 respectively. From the top, can find that the number of layers increases, some correlation values are reduced, indicating that some correlations are filtered out. In contrast, the bottom case illustrates some correlations are enhanced the number of layers increases. These results show that EV-FGN can adaptively and effectively capture important patterns while removing noises learn discriminative model. More visualizations are provided in Appendix H..
Figure 4: The diffusion process of EV-FGN at two timestamps (top and bottom) on COVID-19.
5 CONCLUSION
In this paper, we define a Fourier graph shift operator (FGSO) and construct the efficient edge-varying Fourier graph networks (EV-FGN) for MTS forecasting. EV-FGN adopted simultancously capture high-resolution dencies. This study makes the first attempt to design complex- complex-valued valued feed-forward network in the Fourier space for efficiently computing multi-layer graph convolutions. Extensive experiments demonstrate that EV-FGN achieves state-of-the-ar performances with higher efficiency and fewer parameters and shows high interpretability in graph representation learning. This study sheds light on efficiently calculating graph operations in Fourier space by learning a Fourier graph shift operator.
spatial-temport dependencies and account for time-varying variable depen-

---

Under review as a conference paper at ICLR 2023
REFERENCES
Dimitros Asteriou and Stephen G Hall. Arima models and the box-jenkins methodology. Applied Econometrics, 2(2):265-288) 286, 2011.
Lei Bai, Lina Yao, Can Li, Xianzhi Wang, and Can Wang. Adaptive graph convolutional recurrent network for traffic forecasting. In NewrIPS, 2020.
Shaojie Bai, J. Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional
and recurrent networks for sequence modeling. CoRR, abs/1 803.01271, 2018.
A. Borovykh, S. Bohte, and C. W. Oosterlee. Conditional time series forecasting with convolutional neural networks. arXiv, 2017.
Defu Cao, Yujing Wang, Juanyong Duan, Ce Zhang, Xia Zhu, Congrui Huang, Yunhai Tong, Bixiong Xu, Jing Bai, Jie Tong, and Qi Zhang. Spectral temporal graph neural network for multivariate time-series forecasting, In NeurlPS, 2020.
Yuzhou Chen, Ignacio DomoiiiuzD Baris Coskunuzer, and Yulia Gel. TAMP-s2GCNe Cou- pling awme-aware multipersistence knowledge representation with spatio- supra graph convolutional networks for time-series forecasting. In International Conference on Learning Representations, 2022.
Lu Chi, Borui Jiang, and Yadong Mu. Fast fourier convolution. In NeurlPS, 2020.
George Dasoulas, Johannes F. Lutzeyer, and Michalis Vazirgiannis. Learning parametrised graph shift operators. In ICLR, 2021.
Hoang Anh Dau, Anthony Bagnall, Kaveh Kamgar, Chin-Chia Michael Yeh, Yan Zhu, Shaghayegh Gharghabi, Chotirat Ann Ratanamahatana, and Eamonn J. Keogh. The UCR time series archive. IEEE CAA J. Autom. Sinica, 6(6): 2019.
Ailin Deng and Bryan Hooi. Graph neural network- anomaly detection in multivariate time series. In AAAI, pp. 4027-4035 AAAI Press, 2021.
Fernando Gama, Alejandro Ribeiro, and Joan Bruna. Stability of graph neural networks to relative perturbations. In ICASSP, pp. 9070-9074, 2020.
John Guibas, Morteza Mardani, Zongyi Li, Andrew Tao, Anima Anandkumar, and Bryan Catanzaro.
Adaptive fourier neural operators: Efficient token mixers for transformers. In ICLR, 2022.
Samar Hadou, Charilaos Kanatsoulis, and Alejandro Ribeiro. Space-time graph neural networks. In International Conference on Learning Representations, 2022.
S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computation, 9(8):1735-1784 1997.
R. Hyndman, A. Koehler, K. Ord, and R. Snyder. Forecasting with exponential smoothing. Springer Berlin Heidelberg, 1111077978-3-540-7111888- 2008.
Elvin Isufi, Fernando Gama, and Alejandro Ribeiro. Edgenets: Edge varying graph neural networks. IEEE Transactions Pattern Analysis and Machine Intelligence, 2021.
Y. Katznelson. An introduction to harmonic analysis. Cambridge University Press,, 1970.
Thomas Kipf, Ethan Fetaya, Kuan-Chieh Wang, Max Welling, and Richard 7emel. Neural relational inference for interacting systems. In ICML, volume 80 of Proceedings of Machine Learning Research, pp. 2693-2702. PMLR, 2018.
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In ICLR. 2020.
Johannes Klicpera, Stefan Weißenberger, and Stephan Günnemann. Diffusion improves graph learning. In NewrlPS, pp. 13333-13345, 2019.
10

---

Under review as a conference paper at ICLR 2023
Nikola Kovachki, Samuel Lanthaler, and Siddhartha Mishra. On universal approximation and error bounds for fourier neural operators. J. Mach. Learn. Res., 22:290:1-290:76, 2021a
Nikola B. Kovachki, Zongyi Li, Burigede Liu, Kamyar Azizzadenesheli, Kaushik Bhattacharya, Andrew M. Stuart, and Anima Anandkumar. Neural operator: Learning maps between function spaces. CoRR, abs/2108. 108481, 2021b.
Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long- and short-term temporal patterns with deep neural networks. In SIGIR, pp. 95-104, 2018.
Yaguang Li, Rose Yu, Cyrus Shahabi, and Yan Liu. Diffusion convolutional recurrent neural network: Data-driven traffic forecasting. In ICLR (Poster), 2018.
Zongyi Li, Nikola Borislavov Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhat- tacharya, Andrew M. Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. In ICLR, 2021.
B. Lim and S. Zohren. Time-series forecasting with deep learning: survey. Philosophical Trans- actions of The Royal Society A Mathematical Physical and Engineering Sciences, 379(2194): 20200209, 2021.
Gonzalo Mateos, Santiago Segarra, Antonio G. Marques, and Alejandro Ribeiro. Connecting the dots: Identifying network structure via graph signal processing. IEEE Signal Process. Mag. 36(3): 16-43, 2019.
Aliaksei Sandryhaila and José M. F. Moura. Discrete signal processing graphs. IEEE Trans. Signal Process., 61(7):1644-1656, 2013,
Santiago Segarra, Gonzalo Mateos, Antonio G. Marques, and Alejandro Ribeiro. Blind identification of graph filters. IEEE Trans. Signal Process., 65(5):1146-111) 2017.
Rajat Sen, -s Fu Yu, and Inderjit S. Dhillon. Think globally, act locally: A deep neural network approach to high-dimensional time series forecasting. In NeurlPS, pp. 4838-4888888 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, pp. 1998-6008, 2017.
Jingyuan Wang, Ze Wang, Jianfeng Li, and Junjie Wu. Multilevel wavelet decomposition network for interpretable time series analysis. In KDD, pp. 2437-2446, 2018.
Mark W. Watson. Vector autoregressions and cointegration. Working Paper Series, Macroeconomic Issues, 4, 1993.
Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven C. H. Hoi. Cost: Contrastive learning of disentangled seasonal-tres representations for time series forecasting. In ICLR. OpenReview.net, 2022.
Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. In NeurlPS, pp. 22419-22452 2021.
Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang, and Chengqi Zhang. Graph wavenet for deep spatial-temporal graph modeling. In ISCAI, pp. 1907-1913, 2019.
Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang, Xiaojun Chang, and Chengqi Zhang, Connecting the dots: Multivariate time series forecasting with graph neural networks. In KDD, pp. 753-763, 2020.
Zhangjing Yang, Weiwu Yan, Xiaolin Huang, and Lin Mei. Adaptive temporal- frequeecyy network for time-series forecasting. IEEE Trans. Knowl. Data Eng., 34(4):1576-1559 2022.
Bing Yu, Haoteng Yin, and Zhanxing Zhu. Spatio-temporal graph convolutional networks: A deep learning framework for traffic forecasting. In IJCAI, pp. 3634-3640, 2018a.
Bing Yu, Haoteng Yin, and Zhanxing Zhu. Spatio-temporal graph convolutional networks: A deep learning framework for traffic forecasting. In IJCAL, pp. 3644-3640, 2018b.
11

---

Under review as a conference paper at ICLR 2023
Liheng Zhang, Charu C. Aggarwal, and Guo-Jun Qi. Stock price prediction via discovering multi- frequency trading patterns. In KDD, pp. 2141-2149, 2017.
Yu Zheng, Xiuwen Yi, Ming Li, Ruiyuan Li, Zhangqing Shan, Eric Chang, and Tianrui Li. Forecasting fine-grained air quality based big data. In KDD, pp. 2267-2276, 2015.
Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In AAAI, pp. 11106-11115 2021.
Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. FEDformer: Frequency enhanced decomposed transformer for long-term series forecasting. In ICML, 2022.
12

---

Under review as a conference paper at ICLR 2023
A
NOTATION
Table Notations
X
entire multivariate time series data, X E NXL
X
multivariate time series input under the rolling setting, X E gXT the number of variables of X
N
the number of timestamps of X
the number of timestamps of X
the prediction length of MTS under the rolling setting
the embedding dimension the embedding of X. X E RRxxxxd
X
X
the spectrum of X, A' E CNxTxd
the graph shift operator the supra-graph, 9 {X, S) attributed to X the Fourier graph shift operator
θ
the parameters of the forecasting model
the embedding matrix the variable embedding matrix
du
the temporal embedding matrix
K
the diffusion steps
the kernel function the weight matrix
W
b
the complex bias weights
F
Discrete Fourier transform
F-1
Inverse discrete Fourier transform the forecasting model
K
K-layer EV-FGN
HEV
the edge-varying graph filters
B CONVOLUTION THEOREM
The convolution theorem Katznelson 1970) one the most important property of Fourier transform. It states the Fourier transform of a convolution of two signals equals the pointwise product of their Fourier transforms in the frequency domain. Given a signal x/n] and a filter h[n]. the convolution theorem can be defined follows:
F((x [m]))) F())F(h)
(13)
where (x [m] Dn [| m] denotes the convolution of x and h. (n m) N denotes (n m) modulo N, and F(r) and F(h) denote discrete Fourier transform of 2 [m]] and h[n]. respectively.
C EXPLANATIONS
C.I THE INTERPRETATION OF FREQUENCYY-IVVARIAT FGSO
In addition to reducing parameter volumes and saving computation costs, the frequency-invariant parameterized FGSO is empirically proved effective to improve model generalization. As we mentioned above, we perform EV-FGN over the input embeddings and adopt the frequency-invariant FGSOs. Relatively to directly adopting the frequency-varian FGSOs (C"xaxd), we subtly "factorize" the frequency-varian
parameterization to the time domain (i.e., the embedding matrix & E R=xd) and the frequency domain (i.e., trequency-invariii raw MTS inputs to improve the model learning capability, while we learn the same transformation (FGSO) for all N T frequency points in the frequency domain (similar CNN with shared-weigh
FGSO dd In the time domain, embed the
13

---

Under review as a conference paper at ICLR 2023
convolution kernels filters that slide along input features). Note that the frequency spectrum in the frequency domain has a global view which each frequency point attends to all variables or timestamps. This treatment in EV-FGN guarantees the model capacity and is empirically proved superior over the treatment without embeddings and with frequency-variant FGSO (please refer to the Ablation study for detailed results). Although the frequency-varee more powerful and flexible than the frequency-invariae one, it introduces more parameters in the frequency domain, especially for multi-layer EV-FGN, and may not obtain superior performance due to inadequate training or overfitting.
parameterization may be
C.2 EXPLANATION TO THE EXTENSION OF DEFINITION TO 2D DOMAIN
Recall Eqs. 4 and 5,
Eq. 4 (kernel summation): [(X) x vi [m]]
Eq. 5 (kernel summation): (X)]i Σド= x[i - = (X [ Vi [n]
When we extend the equations to 2D domain, i.e., from [n] to [N] x [T], it means performing a kernel summation/graph convolution over the discrete spatial-temporal space corresponding to all nodes in the supra-graph. Obviously, these computations of the kernel summation can be easily extended to 2D domain. Similarly, according to the convolution theorem,
obtain a 2D-version of Eq. 6:
Eq. 6 (graph convolution): ((X))))== -F(X)) (1)
=
Vi [M] x [T]
with F and FF1 denote the 2D discrete Fourier transform and its inverse, respectively. Accordingly, when extending Definition 2 to 2D domain, JF denotes the 2D discrete Fourier transform. Therefore, perform 2D discrete Fourier transform each discrete
given input embeddings X E R xxxx N spatial-temporal plane of the embeddings to obtain the frequency spectrum, and then feed the frequency input into K-layer EV-FGN followed by two-layer (real-valued) feed-forward networks generate multi-step forecasting (see Section 3.1 for more details). Note that we adopt the frequency- invariant FGSO (d x in the EV-FGN where the feed-forward computations act on the embedding dimension, i.e., d. In addition, when extending Definition 2 to 2D domain, the frequency invariant FGSO invariant to both frequency components derived from the spatial dimension (N) and time dimension (T) respectively.
C.3 INTERPRETATION OF OMITTING THE WRIGHT MATRIX W IN PROPOSITION
This treatment of omitting the weight matrix W in Eq. 9 is feasible since the weight matrix W can be absorbed in the embedding input, precisely the embedding matrix parameters. Note that we feed the input embeddings into EV-FGN (refer to Fig. [ and Section 3.1). In addition, this treatment will not reduce the capability of EV-FGN intuitively since we adopt edge-varying filters in EV-FGN. Note that traditional GCNs adopt different weight matrices (regarding the model capability) but the same GSO (e.g.. adjacency and Laplacian matrices regarding the given graph structure) in different diffusion orders. Differently, there is no pre-given graph structure in MTS forecasting scenarios. therefore we adopt the edge-varying filters, i.c., varying GSOs in EV-FGN, which does not reduce the model capability and achieves desirable performance empirically.
D PROOFS
D. PROOF OF PROPOSITION |
The proof aims to expand the graph convolution corresponding to Hgv using a set of FGSOs in the Fourier space. According to the concise form of Equation ( i.e.,
F(SX) F(X) x. S
(14)
14

---

Under review as a conference paper at ICLR 2023
where FF denotes the discrete Fourier transform (DFT) and omit the weight matrix W for convenience (we can treat W as an identity matrix), it yields:
F(SKSK-1 SoX) = (6
=F Xn Sk
(15)
F(2) Xn So ... SK-1 SK
where {S.) is set of GSOs, and {S.] is a of FGSOs corresponding to ( individually.
Thus, the edge-varying graph filter HEV can be rewritten as
X( F(S,X + S15oX +
= F((xX) F(S + + -(5
(16)
=F(X) xn So + F(X) xn So Xn S + F(X)So Xn .... ...-K-1 Xn SK
Accordingly, we have
K F(X) Xn Suzk
=
s.t. So.k Su Xn ...
SK-1 Xn SK
(17)
Proved.
E COMPARED WITH OTHER NETWORKS
E.1 GNN
Graph Convolutional Networks. Graph convolutional networks (GCNs) depend on the Laplacian eigenbasis to perform the multi-order graph convolutions over given graph structure. Compared with GCNs, EV-FGN as an efficient alternative to multi-order graph convolutions has three main similar costly matrix operations transforms the input into Fourier domain by discrete Fourier transform (DFT) instead of graph Fourier transform (GFT); 2) Explicitly assigning various importance to nodes of the same neighborhood with different diffusion steps. EV-FGN adopts different Fourier graph shift operators S in different diffusion steps corresponding the time-varying dependencies among nodes; 3) EV-FGN invariant to the discretization N, T. It parameterizes the graph convolution via Fourier bases invariant graph structure and graph scale.
differences: 1) No eigendecompositions
required. EV-FGN
Graph Attention Networks. Graph attention networks (GATs) are non-spectral sicention-based graph neural networks. GAT's use node representations calculate the attention weights (i.e., edge weights) varying with different graph attention layers. Accordingly, both GAT's and EV-FGN do not depend on eigendecompositions and adopt varying edge weights with different diffuusinn steps (layers). However, EV-FGN can efficiently perform graph convolutions in the Fourier space. For complete graph, the time complexity of the attention calculation of layers is proportional to Kn² where is the number of nodes, while K K-layer EV-FGN infers the graph structure in Fourier space with the time complexity proportional to n log n. addition, compared with GATs that implicitly achieve edge-varying weights with different layers, EV-FGN adopts different FGSOs in different diffusion steps explicitly.
E.2 FNO
Inspired by Fourier neural operator (FNO) Li et al. (02T) that computes the global convolutions in the Fourier space, we elaborately design EV-FGN to compute the multi-order graph convolutions in the Fourier space. Both FNO and EV-FGN replace the time-consuming graph/global convolutions in time domain with the efficient spectral convolution in the Fourier space according to the convolution theorem.
However, FNO and EV-FGN quite different in the network architecture. As shown in Fig. 5 FNO consists of a stack of Fourier layers where each Fourier layer serially performs 1) DFT to obtain the spectrum of the input, 2) then the spectral convolution in the Fourier space, 3) and finally IDFT to transform the output to the time domain. Accordingly, FNO needs K pairs DFT and IDFT for K Fourier layers in FNO. In contrast, EV-FGN with just pair of DFT and IDFT performs multi-order
15

---

Under review as a conference paper at ICLR 2023
(multi-layer) graph convolutions one fly via stacking multiple FGSOs in Fourier space, as shown in Fig. [
In addition, adaptive Fourier neural operator (AFNO) Guibas al. (2022) is variant of FNO. It a neural unit and used in a plug-and-play fashion as an alternative to self-attention to reduce the quadratic complexity. Similarly, requires a pair of DFT and IDFT to accomplish the self-attention computation. In contrast, our proposed EV-FGN a neural network with a set of FGSOs well-designed connection.
Fourier Layer
Fourier Layer 2
Fourier Layer
F
Figure 5: The simplified structure of FNO derived from Li et al. 2021 F and F denote Fourier transform and its reverse respectively.
F EXPERIMENT DETAILS
F.1 DATASETS
Solar This dataset is about the solar power collected by National Renewable Energy Laboratory. We choose the power plant data points in Florida as the data set which contains 593 points. The data collected from 2006/01/01 2016/12/31 with the sampling interval of every hour.
Wik This dataset contains number of daily views of different Wikipedia articles and is collected from 2015/7/1 to 2016/12/31. It consists of approximately 145k time series and we randomly choose 2k from them as our experimental data set.
Traffid=) This dataset contains hourly traffic data from 963 San Francisco freeway lanes. The traffic data are collected since 2015/01/01 with the sampling interval of every hour.
ECG This dataset is about Elletrocardiogram(ECGG from the UCR time-series classification archive Dau et al. 2019). It contains 140 nodes and each node has a length of 5000.
Electricity This dataset contains electricity consumption of 370 clients and collected since 201 2011//1 Data sampling interval is every 15 minutes.
COVID-19 This dataset is about COVII-19 hospitalization in the U.S. states of California (CA) from 01/02/2020 to 31/12/2020 provided by the Johns Hopkins University with the sampling interval of every one day.
METR-LA This dataset contains traffic information collected from loop detectors in the highway of Los Angeles County. contains 207 sensors which is from 01/03/2012 to 30/06/2012 and the data sampling interval is every 5 minutes.
htg.::/cinncccccceoooooorraaaaar forecasting/datt
BCG5000
httes//githnbb.coo///ggggnn/n//
16

---

Under review as a conference paper at ICLR 2023
F.2 BASELINES
VAR Watson 1993p: VAR classic linear autoregressive model. We use Statsmodels library (htpps: / www. st stat ataatmmmollsss. org) which is python package that provides statistical compu- tations to realize the VAR.
DeepGLO Sen et al. (219)): DeepCiLO models the relationships among variables by matrix factor- ization and employs temporal convolution neural network to introduce non-linear relationships. We download the source code from: https: /t/::::/ghtti com/ rajat sen deepglol We use the default setting as our experimental settings for wiki, electricity and traffic datasets. For covid datasets, the vertical and horizontal batch size is set to 64, the rank of global model is set 64, the number of channels is set to [32, 32, 32, 1]. and the period is set to 7.
LSTNet Lai et al. 2018): LSTNet uses CNN to capture inter- relationships and RNN to discover long-term patterns. We download the source code from: https: / / wiithuu.... com/ In our experiment,
the default settings where the CNN hidden units is 100, the kernel size of the CNN layers is 4, the dropout is 0.2, the RNN hidden units is 100, the number of RNN hidden layers is the learning rate is 0.001 and the optimizer Adam.
w...uooknnL..TTE
TCN Bai et al. 2018): TCN is a causal convolution model for regression prediction. We download the source code from: https: / github. com/ wacccussla//ooo We utilize the same configuration as the polyphonic music task exampled in the open source code where the dropout is 0.25, the kernel size is 5, the hidden units is 150, the number of levels 4 and the optimizer is Adam.
Reformer Kitaev et al. 2020): Reformer combines the modeling capacity of a Transformer with an architecture that can be executed efficiently on long sequences and with small memory use. We download the source code from: https: : //githbb com/ ctw/thumm thuml /Aatmmll/Aoforeerr We use the
recommended settings as the experimental settings.
Informer Zhou et al. 2021): Informer leverages efficient self-attention mechanism to encode the dependencies among variables. We download the source code from: https: h/pp:://// thub. com/ Shouhaoyi/Informerr/
We use the default settings as our experimental settings where the dropout is 0.05, the number of encoder layers is 2, the number decoder layers 1, the learning rate is 0.0001, and the optimizer is Adam.
Autoformer Wu et al. 2021]; Autoformer proposes a decomposition architecture by embedding the series dccomposition block an inner operator, which can progressively aggregate the long-term trend part from intermediate prediction. We download the source code from: tttps: : / thub com/thuml ott.:Atttoooooooooooo We use the recommended settings our experimental settings with encoder layers and decoder layer.
FEDformer Zhou al. (0022): FEDformer proposes attention mechanism with low-rank approxi- mation in frequency and a mixture of experts decomposition to control the distribution shifting. We download the source code from: https: /gtbbbb.ttttuuggo thub. com/ MAZi qing / FEDformerr We FEB-f as the Frequeney Enhanced Block and select the random mode with 64 as the experimental mode.
SFM Zhang et al. (2017): : On the basis of the LSTM model, SFM introduces series of different frequency components in the cell states. We download the source code from: / thub com/ 2om/23315651 0111166536600 wtt.///teee-reeeeeceeeeeeeooooe
precccc-peeiicoo ct inn. We use the de- fault settings as the authors recommended where the learning rate is 0.01, the frequency dimension is 10, the hidden dimension is 10 and the optimizer is RMSProp.
StemGNN Cao et al. 2020):: StemGNN leverages GFT and DFT to capture dependencies among variables in the frequency domain. We download the source code from: https:: httts:://ittub. com/ microsoft// emGNN We use the default setting of stemGNN as our experiment setting where the optimizer is RMSProp, the learning rate is 0.0001, the stacked layers is 5, and the dropout rate is 0.5.
MTGNN Wu al. (2020): MTGNN proposes effective method exploit the inherent dependency relationships among multiple time series. We download the source code from: https: //ththb/t/ Com/nnzhan/MTann Because the experimental datasets have no static features, set the
17

---

Under review as a conference paper at ICLR 2023
parameter load static feature false. We construct the graph by the adaptive adjacency matrix and add the graph convolution layer. Regarding other parameters, we adopt the default settings.
Graph WaveNet Wu al. (2019) GraphWaveNet introduces adaptive dependency matrix learning to capture the hidden spatial dependency. We download the source code from: Intpp:: /ggitttt Since our datasets have no prior defined graph structures, use only adaptive adjacent matrix. We add a graph convolution layer and randomly initialize the adjacent matrix. We adopt the default setting as our experimental settings where the learning rate is 0.001, the dropout is 0.3, the number of epoch is 50, and the optimizer is Adam.
com/nnzhan Graph-WaveNee
AGCRN (Bii al. 2020) AGCRN proposes data-adaptive graph generation module for discovering spatial correlations from data. We download the source code from: https: //t::://ttti httptttttbtu.///./// com/ LeiBAI/AGCRN We use the default settings as our experimental settings where the embedding dimension is 10, learning rate is 0.003, and the optimizer is Adam.
TAMP-S2G:CNets Chen al. y022): TAMP-S2GCNer explores the utility of MP to enhance knowl- edge representation mechamisms within the time aware DL paradigm. We download the source code from: https://wwwt dropbox com/sh/ ja5 cotm//ah/r0aaad111ttteeee////// (ttp://eeeddeeo/AAAA--------- eete://AAAnnneeiViieeeee LOAOsNa 2d ?d1=0, TAMP-S2GCNets requires predefined graph topology and we use the Cali- fornia State topology provided by the source code as input. We adopt the default settings as our experimental settings COVID-19.
DCRNN Li et al. (2018: DCRNN uses bidirectional graph random walk to model spatial dependency and recurrent neural network to capture the temporal dynamics. We download the source code from We use the default settings as our experimental settings with the batch size 64, the learning rate is 0.01, the input dimension is 2 and the optimizer is Adam. DCRNN repuires a pre-defined graph structures and we use the adjacency matrix as the pre-defined structures provided by METR-LA dataset.
https://gi githtb.thuuuu.. com/ 1 Eayyaguang/DCRRI
STGCN Yu et al. (2018)): STGCN integrates graph convolution and gated temporal convolu- tion through spatial-temporal convolutional blocks. We download the source code from[https] //github. com cwm/Verii Weeritsstinn.. Bwt.iiii/TTNNNCCCCC IICNNNIJCIII 18) We use the default settings as our exper- imental settings where the batch size 50, the learning rate is 0.001 and the optimizer is Adam. STGCN requires pre-defined graph structures and we leverage the adjacency matrix as the pre-
defined structures provided by METR-LA dataset.
CoST Woo et al. (2022): ; CoST separates the representation learning and downstream forecasting task and proposes a contrastive learning framework that learns disentangled season -rend representations for time series forecasting tasks. We download the source code from: https: : // / g titbb.b.. com/
sales force/ CoST We set the representation dimension to 320, a learning rate to 0.001 and the batch size to 32. Inputs are min-max normalization, we perform 70/20/10 train/validationntess split for METR-LE -LA dataset and 60/20/20 for COVID-19 dataset.
F.3 EVALUATION METRICS
We MAE (Mean Absolute Error), RMSE (Root Mean Square Error), and MAPE (Mean Absolute Percentage Error) as the evaluation metrics in the experiments.
Specifically, given the groudtruth x(+1it+r E Nxr and the predictions E++1147 E N for future steps at timestamp f, the metrics are defined as follows:
N
MAE
Zij - Azyl
(18)
TN
1
RMSE
(xi R.1y)
(19)
j=1
N
1 NN
Iij Rij
MAPE
x 100%
(20)
Iij
j=1
with mi E Xt+lit+r and Byg &t11itt
18

---

Under review as a conference paper at ICLR 2023
F.4 EXPERIMENTAL SETTINGS
We summarize the implementation details of the proposed EV-FGN as follows. Note that the details of the baselines are introduced in their corresponding descriptions (see Section F.2
Network details. The fully connected feed-forward network (FFN) consists of three linear transfor-
mations with Leaky RU activations in between. The FFN formulated follows:
X, = Leaky & W1 + bT)
X₂ LeakyReLU(XI/ W₂ + b2)
(21)
X X2W3 + b3
where Wi E (RTd)xdf" W2 E "df and W3 E R4 are the weights of the three layers respectively, and b E Rdf b2 Rdd and by R the biases of the three layers respectively. Here, df and df'n are the dimensions of the three layers. In addition, we adopt ReLU activation function in Equation
Table 6: Dimension settings of FFN on different datasets.
Datasets
Solar
Wiki
Traffic
ECG
Electricity
COVID-19
META-LR
1
6
2
2
*
4
df
64
64
64
64
64
256
64
256
256
256
256
256
512
256
denotes that we feed the original time domain representation to FFN without the dimension reduction.
Training details. We carefully tune the hyperparameters, including the embedding size, batch size, df and dn the validation set and choose the settings with the best performance for EV-FGN on different datasets. Specifically, the embedding size and batch size are tuned over {32, 64, 128, 256, 512) and {2, 8, 16, 32, 64, 128) respectively. For the COVID-19 dataset, the embedding size is 256, and the batch size is set to 4. For the Traffic, Solar and Wiki datasets, the embedding size is 128, and the batch size is the embedding size is 128, and the batch size is set to 32. Note that the supra-graph connecting all nodes is cunnecco graph, indicating that any spatial order is feasible. Therefore, although we perform 2D DFT in EV-FGN, none spatial order in the spatial space is necessary for performing EV-FGN, and we directly adopt the raw dataset for experiments.
2. For the METR-LA, ECG and Electricity datasets,
To reduce the number of parameters, we adopt linear transform to map the original time domain representation X E IR NxTxd to low-dimensional tensor X E Nxlxd with < T. We then reshape X (x and feed it to FFN. We perform grid search on the dimensions of FFN, i.e., fm and dfd", over (32, 64, 128, 256, 512) and tune the intermediate dimension over {2, 4, 6, 8, 12). The settings of the three hyperparameters over all datasets are shown in Table 6 Finally, we set the diffusion step =3
3 for all datasets.
F.5 DETAILS FOR VISUALIZATION EXPERIMENTS
To verify the effectiveness of EV-FGN in learning the connected supra- supra-graph. we obtain the output of EV-FGN as the node representation, denoted as R = x IR-V x xd with inverse discrete Fourier transform (IDFT) and K-layer EV- FGN Kg. Then, we visualize the adjacency matrix A calculated based the flatten node representation R e NTxx, formulated as A RRT NT to show the variable correlations. Note that A normalized via A/ max(A) ()). Since is not feasible to directly visualize the huge adjacency matrix A of the supra-graph, visualize its different subgraphs in Figures 3, 4, and 10 better verify the learned
dependencies on the fully-
information on the supra- supra-graph from different perspectives.
Figure 3: On the METR-LA dataset, average its adjacency matrix A over the temporal dimension (i.e.. marginalizing T) to A' e NNN, Then, we randomly select 20 detectors out of all N 207 detectors and obtain their corresponding sub adjacency matrix (IR 20x00 from A' for visualization.
19

---

Under review as a conference paper at ICLR 2023
We further compare the sub adjaceney with the real road map (generated by the google map tool) to verify the learned dependencies between different detectors.
Figure 4. On the COVID-19 dataset, randomly choose 10 counties out of N 55 counties and obtain their four sub adjacency matrices of four consecutive days for visualization. Each of the four sub adjacency matrices IR 10x embodies the dependencies between counties in one day. Figure reflects the tiry-ng dependencies between counties (i.e., variables).
calculate four adjaceney matrices based on the input X' of EV-FGN and the outputs of each layer in EV-FGN, i.e., Ai, X2, X3. Following the way of visualization in Figure visualization. Figure 5 shows the effects of each layer of EV-FGN in filtering or enhancing variable correlations.
Figure 5. Since we adopt 3-layer EV-FGN,
select 10 counties and two timestamps on the four adjacency matrices for
Figure 10. We select counties and visualize the correlations between 12 consecutive time steps for each selected county respectively. Figure 10 reflects the temporal correlations within each variable.
G MORE RESULTS
Table 7: Performance comparison under different prediction lengths on the COVID-19
19 dataset.
Length
6
12
Metics
MAE
RMSE
MAPB(S)
MAE
RMSE
MAPE(S)
MAE
RMSE
MAPB(S)
MAE
RMSE
MAPE/S)
Grapl Wa et
2019
0.092
0.129
53.00
0.133
0.179
66.11
0.171
0,225
80.91
0.201
0.255
100.53
SlemON Cao
0247
0,318
99.98
0,344
01429
125.81
0.359
0.442
131.14
0.421
0.508
141.01
AGCRN Bii a
0.130
0.172
68.64
0.171
0.188
79.29
0.224
0.277
113:42
0.254
0.309
125.43
MIGNN Wa
0.276
0.379
91.42
0.446
0.513
133.49
0.484
0.548
139.52
0.394
0.488
$8.13
TAMP-S2GUNeS
Neseehhseeeeeeeee Lhe al. 2022
0.140
0.190
50001
0.150
0.200
55.72
0.170
0,230
71.78
0.150
0.230
65.76
CoST Woo
0.122
0.246
68.74
0.157
0.318
72.84
0.183
0.364
77.04
0.202
0.377
80.81
EV-PGN(ours)
0.011
0.103
61.02
0.093
68.72
0.109
0.148
69.59
0.124
0.164
72.57
Improvement
22.8%
20.2%
30.1%
26.8%
35.9%
35.7%
3.1 %
31.1%
28.7%
To further evaluate the performance of our model EV-FGN in multi-step forecasting, we conduct more experiments on the COVID-19, Wiki, ECG, and METR-LA datasets, respectively. Note that the COVID-19 and METR-LA datasets have predefined graph topologies. We compare EV-FGN with other GNN-based MTS models (including StemGNN, AGCRN, Graph WaveNet, MTGNN and HAMP-S2GCNett and representation learning model (CoST) on the COVID-19 dataset under different prediction lengths, and the results are shown in Table 7 From the table, we can find that EV-FGN achieves the best MAE and RMSE on all the prediction lengths. On average, EV-FGN has 30.0% and 27.9% improvement TAMP-S2GCNets. Among these models, FAMP-S2GCNets requiring a pre-defined graph topology achieves competitive performance since it enhances the resultant graph learning mechanisms with a multi-persistence However, it constructs the graph in the spatial dimension, while our model EV-FGN adaptively learns supra-graph connecting any variables at any two timestamps, which is effective and more powerful to capture high-resolution spatial-temporal dependencies.
MAE and RMSE respectively over the best baseline, i.e.,
Table 8: Performance comparison under different prediction lengths on the Wiki dataset.
Length Metries
3
6
g
12
MAE
RMSE
MAPE(S)
MAE
RMSE
MAPE(S))
MAE
RMSE
MAPE(S)
MAE
RMSE
MAPE(S)
Wavee Wu et al.
2019
0.061
0.105
138.60
0.061
0.105
135.32
0.061
0.105
132.52
0.061
0.104
136.12
StemGN Cao clal
A140
0.157
0.236
89.00
0.159
0.233
98.01
0.232
0.311
142.14
0.220
0.306
125.40
AGCRN al
20:2
0.043
0.077
73.49
0.044
0.078
80.44
0.045
0.079
81.89
0.044
0.079
78.52
MTGNN Will et
A000
0.102
0.141
123.15
0.091
0.135
91.75
0.074
0.120
85.44
0.101
0.140
122.96
Informer Zhou of
0.053
0.089
85.31
0.054
0.099
84.46
0.059
0.095
95.30
0.059
0.095
95.00
EV-PGN(oursi
0,040
0.075
58.18
0.641
0.075
60.43
0.041
0.076
60.95
0.042
0.077
62,62
Improvement
7.0%
2.6%
20.83%
6.8%
3.8%
24.9%
8.9%
3.8%
25.6%
4.5%
2.5%
20.3%
In addition, we compare our model EV-FGN with five neural MTS models (including StemGNN, AGCRN, Graph WaveNet, MTGNN and Informer) Wiki dataset under different prediction lengths,
20

---

Under review as a conference paper at ICLR 2023
and the results are shown in Table 8 From the table, observe that EV-FGN outperforms other models on MAE, RMSE and MAPE metries for all the prediction lengths. On average, EV-FGN improves MAE, RMSE and MAPE by 6.8%, 3.2% and 22.9%, respectively. Among these models, AGCRN shows promising performances since it captures the spatial and temporal correlations adaptively. However, it fails simultaneously capture spatial-temporal dependencies, limiting its forecasting performance. In contrast, our model learns a supra-graph to capture comprehensive spatial-temporal dependencies simultancously for multivariate time series forecasting.
Table 9: Performance comparison under different prediction lengths on the METR LA dataset.
Horizon
3
6
12
Metrics
MAE
RMSE
MAPE(S)
MAE
RMSE
MAPE(S)
MAE
RMSE
MAPE/S)
MAE
RMSE
MAPE(S)
DCRNNLier
0.160
0.204
80.00
0.191
0.243
85.15
0.216
0.269
85.72
0.291
STGCN al
20180
0.058
0.133
59.02
0.080
0.177
60.67
0.102
0.209
62.08
0.128
0.238
63.8]
Gragh WaraNe
2019
0.180
0.366
21.50
0.184
0.375
22.95
0.196
0.382
23.61
0.202
0.386
24.14
MTGNN Wu dral
2101
0.135
0.294
17.99
0.144
0.307
18.82
0.149
0.328
19.38
0.153
0.316
19.92
StemGNiy Lib eq
2000
0.052
0.115
86.39
0.069
0.141
87.71
0,080
0.162
89.00
0.093
0.175
90.25
AGCRN
0.131
0.165
27.62
0.188
0.204
31.73
Informer Zhou ai 2021
0.076
0.141
69.90 99.96
0.088
0.163
70.94
0.096
0.17%
72:26
0.100
0.190
72.54
CoS 1 Wotrer
0.064
0:118
88.44
0.077
0:141
89.63
0.088
0.159
90.56
0.097
0.121
91.42
EV-PGN(ours)
0.113
86.30
0.140
87.97
0.076
0.159
88.99
0.168
89.89
Imgrovement
3.8%
1.7%
4,3%
0.7%
5.0%
9.7%
3.5%
Finally, we compare our model EV-FGN with seven neural MTS models (including STGCN, DCRNN, StemGNN, AGCRN, GraphWaveNet MTGNN, Informer and CoST) on the METR-LA dataset, and the results are shown in Table 9 On average, we improve 5.7% on MAE and 2.5% on RMSE. Among these models, StemGNN achieves competitive performance because it combines GFT to capture the spatial dependencies and DFT to capture the temporal dependencies. However, it is also limited to simultaneously capturing apatial-teppore dependencies. CoST learns disentangled seasonal -rend representations for time series forecasting via contrastive learning and obtains competitive results. But, our model still outperforms CoST. Because, compared with CoST, our model not only can learn the dynamic temporal representations. but also capture the discriminative spatial representations. Besides, STGCN and DCRNN require pre-defined graph structures. But StemGNN and our model outperform them for all steps, and AGCRN outperforms them when the prediction lengths are 9 and 12. This also shows that a novel adaptive graph learning can precisely capture the hidden spatial dependency. In addition, we compare EV-FGN with the baseline models under the different prediction lengths on the ECG dataset, as shown in Fig. 6 It reports that EV-FGN achieves the best performances (MAE, RMSE and MAPE) for all prediction lengths.
0.32
0.14
0.30
0.08
0.10
20
0.00
10
(redkTiC invele
cendiction lnwdft
(a) MAE
(b) RMSE
(c) MAPE
Figure 6: Performance comparison in different prediction lengths on the ECG dataset.
H MORE ANALYSES
H.1 ANALYSES
Parameter Analysis. We evaluate the forecasting performance of our model EV-FGN under different diffusion steps on the COVID-19 dataset, as illustrated in Table 10 The table shows that EV-FGN achieves increasingly better performance from K to K 4 and achieves the best results when K = 3. With the further increase of K. EV-FGN obtains inferior performance. The results indicate that oigh-order diffusion information is beneficial for improving the forecasting accuracy, but the
21

---

Under review as a conference paper at ICLR 2023
Table 10: Performance different diffusion steps the COVID-19 dataset.
K=1
K=2
K=3
K=4
MAE
0.136
0.133
0.129
0.132
RMSE
0.181
0.177
0.173
0.176
MAPE(%)
72.30
71.80
71.52
72.59
6.0000
MAP
113
0.078
21.43
00790
0.0700
31.03
0.785
124
30095
00794
10
15
20
22
30
120
Input kng
urrlssdingg...
Figure 7: Influence of input length.
Figure 8: Influence of embedding size.
diffusion information may gradually weaken the effect or even bring noises to forecasting with the increase of the order.
In addition, we conduct extensive experiments on the ECG dataset to analyze the effect of the input length and the embedding dimension d, shown in Fig. 7 and Fig. respectively. Fig. 7 shows that the performance (including RMSE and MAPE) of EV-FGN gets better as the input length increases, indicating that EV-FGN can learn comprehensive supra-graph from long MTS inputs capture the spatial and temporal dependencies. Moreover, Fig. Bishow that the performance (RMSE and MAPE) first inereases and then decreases with the increase of the embedding size, which is attributed that large embedding size improves the fitting ability of EV-FGN but it may easily lead to the overfitting issue especially when the embedding size is too large.
180
0.20
0.25
340
0.35
8 0.20
0.30
90
0.00
300
350
400
450
500
300
450
Mumber of
Mamber Nettt
(a) MAE
(b) RMSE
(c) MAPE
Figure 9: The sensitivity between MAE, RMSE, MAPE and number of nodes the Wiki dataset.
We further conduct experiments on the Wiki dataset to investigate the performance of EV-FGN under different graph sizes. The results are shown Fig. D where Fig. [2) Fig. 9( and Fig, [c) show MAE, RMSE and MAPE the different number of nodes, respectively. From these figures, observe that EV-FGN keeps a leading edge over the other ofate-of-the-art MTS models as the number of nodes increases. The results demonstrate the superiority and scalability of EV-FGN large scale datasets.
Ablation Study. We provide more details about each variant used in this section and Section
w/o Embedding. A variant of EV-FGN feeds the raw MTS input instead of its embeddings into the graph convolution in the Fourier space.
22

---

Under review as a conference paper at ICLR 2023
Table 11: Ablation studies the COVID-19 dataset.
metrics
w/o Embedding
w/o Dynamic Filter
w/o Residual
w/o Summation
EV-FGN
MAE
0.157
0.138
0.131
0.134
0.129
RMSE
0.203
0.180
0.174
0.177
0.173
MAPE(%) 76.91
74.01
72.25
72.57
71.52
w/o Dynamic Filter. A variant of EV-FGN uses the same FGSO for all diffusion steps instead of applying different FGSOs in different diffusion steps. It corresponds to a vanilla graph filter.
w/o Residual. A variant of EV-FGN does have the layer output, i.e., X, in the summation.
w/o Summation. variant of EV-FGN adopts the last order (layer) output Xx the final frequency output of the EV-FGN.
We conduct another ablation study on the COVID-19 dataset to further investigate the effects of the different components of our EV-FGN. The results are shown in Table [T] which confirms the results in Table 4 and further verifies the effectiveness of each component in EV-FGN. Both Table [T and Table report that the embedding and dynamic filter in EV-FGN contribute more than the design of residual and summation to the state-of-the ff-thh-art art performance of EV-FGN.
N=7
(b) N=18
(c) N=24
N=29
(c) N=38
( N=45
N=46
(h) N=55
Figure 10: The temporal adjacency matrix of eight variables on COVID-19 dataset.
Ordering of the time series. Note that the supra-gras graph connecting all nodes is a fully-connected graph, indicating that any spatial order is feasible. Then how could we perform 2D DFT achieve the graph convolution? First, the graph convolution on the supra-graph can be viewed as kernel summation (cf. Eq.4, i.e., O(X)ii
Σ x(j) j]. Vj E [n]) and does not depend on specific spatial order. From Eq.4 to Eq.5, we extend the kernel summation
kernel integral in Green's 's kernel x) x(i - j)). According the convolution theorem, we can reformulate the graph convolution with continuous Fourier transform (i.e., Eq ((X) = (F(X) (i), Vi E D) and then apply Eq.6 to the finite discrete spatial-temporal space (i.e., the supra- supra-graph). Accordingly, can obtain the Definition and Eq.7 reformulating the graph convolution with 2D DFT.
the continuous spatial-tense temporal space and introduce special kernel, i.e., the
To understand the learning paradigm of EV-FGN, we can recall the self-attention mechanism as an analogy that does not need any assumption on datapoint order and introduce extra position embeddings for capturing temporal patterns. EV-FGN performs 2D DFT on each discrete x T -patial-temporal plane of the embeddings X instead of the raw data X. A key insight underpinning EV-FGN is to introduce variable embeddings and temporal position embeddings to equip EV-FGN with sense of variable correlations (spatial patterns and temporal patterns).
23

---

Under review as a conference paper at ICLR 2023
To verify the claim, conducted experiments the dataset ECG, Specifically, randomly shuffle the order of time-series variables the raw data five times and evaluate our EV-FGN on each shuffled data. The results are reported in the following table. Note that also conduct shuffling experiments on temporal order via performing the same shuffling scheme over each window-sized time-series input, which will get the same conclusion.
Table 12: Five-round results randomly shuffled data the dataset ECG.
Metric
R1
R2
R3
R4
R5
Raw
MAE
0.052
0.053
0,053
0,053
0,052
0.052
RMSE
0.078
0.078
0.078
0.078
0.078
0.078
MAPE(%) (%)
10.95
10.98
11.02
10.99
10.99
11.05
H.2 VISUALIZATIONS
To demonstrate the ability of our EV-FGN in jointly learning spatial-temporal dependencies, visualize the temporal adjacency matrix of different variables. Note that the spatial adjacency matrices of different days are reported in Fig. 3 Specifically, we randomly select counties from the COVID- 19 dataset and calculate the correlations of 12 consecutive time steps for each county. Then we visualize the adjacency matrix via heat map, and the results shown in Fig. 10 where N denotes the index of the country (variable). From the figure, we observe that EV-FGN learns clear and specific temporal patterns for each county. These results show that our EV-FGN can not only learn highly interpretable spatial correlations (see Fig. 2 and Fig. 3 but also capture discriminative temporal patterns.
24


Reviews:
[{"review_id": "ohDrXGH1Z2", "reviewer": "Reviewer_zvHq", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes using Fourier Neural Network (FNO) to combine Graph Shift Operator (GSO), to enable multivariate forecasting.\nThe architecture first obtains a Fourier transform of embedding of input and then applies it to the proposed operator FGSO, which mainly is consisted of FNO. The authors compare the proposed approach with other Graph based and transformer based forecasting approaches (including ablated versions of the proposed approach) on 7 datasets. They demonstrate competitive performance of the proposed approach. ", "summary_of_the_review": "Overall I think the idea is interesting and in particular, the direction of making multivariate forecasting using FNO is interesting even if it's a simple extension. However, it could be useful to point out why FNO is useful and compatible with GSO and contrast it to related alternate approaches for multivariate forecasting, such as Fedformer. There are several concerns about the method and questions that I feel need to be addressed, i.e., specifics of the method so it could be understood by the readers. Without additional clarifications, I am leaning on the side of rejection. I will wait for the author to clarify before making my final decision.\n\n\n===============================\n\nAfter rebuttal:\nThe authors made additional clarifications and addressed some concerns. I raised my score from 5 to 6", "strength_and_weaknesses": "## Strength\n\n1. The paper is well-written. I liked how the authors posed the idea of FNO as a simple add-on of GSO to MTS. \n2. A decent collection of datasets and experiments were performed, and an extensive set of graph neural net based and transformer based forecasting methods were compared.\n3. Additional useful ablation study, hyperparameter sensitivity study, and visualization result analyses were also provided. This provides more interesting information and an understanding of the proposed approaches.\n\n## Weakness:\n\n1. Novelty is somewhat limited. As pointed out there are several works learning graph structure as part of forecasting, this work is a straightforward extension of the original GSO by replacing the original operations with FNO. From the novelty point of view, FNO is a recently proposed method for solving PDEs, and Fedformer (ICML 2022, authors already cited and compared) extends FNO to long-term forecasting. I do not recall any work that combines FNO and graph structure to perform multivariate time series forecasting, but the combination of GSO and FNO in this work is simple. \n\n2. Overall, I found the article tricky to follow. I don’t find the plots in Figure 1 very insightful and it is quite hard to see any informative insights regarding EV-FGN in my opinion. The caption is just \"The network architecture of our proposed model\". The detailed FGSO is missing in Figure 1, which makes the reader confused about how FNO is integrated with GSO. Based on the codes provided by the authors, I don't see any difference between FGSO and FNO except for the real and imaginary concatenation parts. It seems to me the concatenation operation is different from the proposed EV-FGN layer in equation 12. Please clarify this point. I feel that more need to be said about architecture and how it numerically affects the type of relations that it is capable of representing, i.e., GSO with and without FNO. Also, \n\n3. The high efficiency of FGSO is not obvious. First, I agree the space-invariant property of FGSO, but the claim of scale-free parameters in the Sec 3 remark is unfair. The complexity of FNO is largely determined by the grid size because it is originally proposed to solve PDEs. Instead, the graph size of MTS considered in this paper is fixed. Directly comparing a method solving PDEs and another solving MTS is unfair. Also, it seems to me the complexity of FNO is not $nd^2$. Numerically, based on table 3, GraphWaveNet trained on traffic dataset has a parameter size of 280860 and takes 105.38 seconds per epoch. On the other hand, EV-FGN has a much smaller parameter size but takes a relatively similar amount of time per epoch (99.25 sec). Could you explain why this happen？ I believe the main computation overhead comes from the Fourier transform and its inverse. \n\n## Questions\n1. Please use notation consistently throughout the paper. For example, in Sec 3.1, it says 'Given input data $X \\in R^{N \\times T}$', '2D discrete Fourier transform (DFT) is applied on the embedding of $X \\in R^{N \\times T \\times d}$', but in definition 2, the input size of  $X$ becomes to ${n \\times d}$. Why is lowercase $n$ in definition 2 and why is not  ${n \\times t \\times d}$? \n2.  The claim that  Fourier Graph Shift Operator captures high-resolution spatial-temporal dependencies is inaccurate or unclear. The Fourier layer in the Fourier Neural operator on its own only uses low-frequency representations, and thus loses higher-frequency modes. The biased term $W$ might help to recover the higher Fourier modes to some extent, but it can not capture high freq features. This is saying adopting FNO is not to capture high-resolution features. If authors regard 'any two variables at any two timestamps' as high-resolution, I would suggest clarifying high-res in the introduction. \n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "FRIcYOKi2W", "reviewer": "Reviewer_VMXz", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper adaptively learns a supra-graph, representing non-static correlations between any two variables at any two timestamps, to capture high-resolution spatial-temporal dependencies, and define FGSO that has the capacity of scale-free learning parameters in the Fourier space. Accordingly, authors construct a complex-valued feed forward network, dubbed as Edge-Varying Fourier Graph Networks (EV-FGN), stacked with multiple FGSOs to perform high-efficiency multi-layer graph convolutions in the Fourier space. ", "summary_of_the_review": "The empirical results are good, while the analytical results are limited.", "strength_and_weaknesses": "The results are adequate in various datasets and look great. The training details are comprehensive. The equations are proved clearly in the paper.\n\nBut The idea seems limited, GSO and MTS with Fourier transform existed before, the paper just extended to other latent space such as Fourier space. The motivation of this is not convincing enough.\n\nQuestions\n1.\tWhy the results on MAPE are higher than other baselines on several datasets?\n2.\tIn Table 3, the authors compare parameter efficiency with old baselines, why not compare with recent baselines from 2021 such as FEDformer?\n3.\tFigure 1 can not visualize the proposed method clearly.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "CpJ-DbGmVO", "reviewer": "Reviewer_jKjm", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a method for discovering the underlying correlations between variables in a temporal system which drive its evolution. Recent works have applied GNNs to model these relationships, but several of these approaches have employed a static graph over time failing to capture the potential evolution of this dependency structure. To address this, the proposed approach utilizes an all encompassing supra-graph that jointly represents relationships between all variables at all timesteps. Towards this, the notion of Fourier Graph Shift Operator is introduced which performs graph convolution in the frequency domain. This representation is exploited in devising a frequency-invariant convolution operation that can avoid the $n^2$ node complexity associated with a dense graph and effectively reduces\n it to $n \\cdot \\log n$. Edge-varying diffusion networks are introduced for modelling higher-order diffusion over multiple neighbourhood sizes and a corresponding Fourier representation is introduced. ", "summary_of_the_review": "The paper proposes a novel approach for performing graph convolution in the Fourier space at a reduced computational cost. This speed-up allows for joint modelling of spatio-temporal interactions. The flexibility of the introduced parametrization, however, is not investigated, and consequently there is a lack of theoretical understanding of what the approach can actually learn. Experiments show minor improvements over the considered baselines, but I think overall these are not conclusive under the given experimental setting. ", "strength_and_weaknesses": "### The paper has several strengths:\n- Efficient GNN parametrization in the Fourier domain resulting in reduced computational complexity compared to naive graph-based approaches for MTS modelling.\n- Consequently, the proposed algorithm allows for joint modelling of spatial and temporal variables, that would be prohibitive otherwise due to computational restrictions.\n- Experiments show SOTA performance on a variety of datasets with better computation time and reduced parameter count.\n- Ablation studies demonstrate the effectiveness of each proposed model component.\n\n### It also has some weaknesses in my opinion:\n- The clarity of the formulation is often obfuscated by confusing terminology.\n\nFor example, at the end of page 4: \"Accordingly, we can parametrize FGSO with a complex-valued matrix which is space-invariant...\"\nThis sentence seems misleading. The same term is used previously to refer to a regular GSO, in which case it meant that the kernel function $\\kappa$ is shared across nodes $[n]$. However, the invariance in this case is with respect to the frequency input in the Fourier domain. Therefore, it is not space but frequency-invariant. It seems to me that this is actually the point that allows the convolution operator to be non-static in time since frequency-invariance in the Fourier domain still allows the convolution to be temporally varying in the input domain. This point could be emphasized more.\n\nSimilarly, I was confused by the extension of the argument by considering a 2D domain (i.e. from $[n]$ to $[N] \\times [T]$). The authors say that \"we can extend Definition 2 to a 2D discrete space\". How would that look like? In that case, would the proposed parametrization invariant be invariant to both frequency components?\n\nIn Proposition 1, the dimensions associated with the $S_k$ and $\\mathcal{S}_k$ operators seem to be confusing. $S_k$ is defined to be $n \\times n$, while $\\mathcal{S}_k$ is defined as $n \\times d \\times d$. What is the relation between the two? \n\nOn a related note, the left-hand side of equation (9) should probably read as $H_{EV} X W$ since (I assume) that it is the edge-varying counter part of equation (7).\n\n\n- No interpretation is provided for the frequency-invariant parametrization\n\n Related, I was somehow lacking an interpretation of what this shared parametrization across frequencies means with respect to the convolution operation in the input domain. Although it allows for a non-static temporal representation, it clearly places a big restriction on the possible spatio-temporal interactions that it can represent. I guess my worry is that it might reduce to something straightforward in the input domain, which is only obfuscated by the dual formalism. What do we get when we invert this? What's the range of operations that it can represent? These are questions not at all addressed at the moment.\n\n- Experimental improvements are often marginal\n\nIn the experiments, the improvement is often marginal over the baselines. Further, in Appendix E.4 the authors report that they carefully fine-tune the hyperparameters of their own model. However, for the baselines in Appendix E.2, they report the use recommended default settings only. Hence, significantly more effort is spent on tuning their model compared to zero tuning on the baselines, which makes the experimental results inconclusive. I would be more convinced if rather than using default settings, they also fine-tuned the baseline settings at least with respect to the hyperparameters deemed significant by the respective authors of these baselines.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "2V09nfRgSfy", "reviewer": "Reviewer_cY1E", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work focuses on multivariate time series forecasting problems. To capture the  ever-changing correlations over time series data, this work defines a Fourier graph shift operator and constructs the efficient edge-varying Fourier graph networks to formulate spatial temporal dependencies for the varying data. Extensive experimental results demonstrate the effectiveness of the proposed method. \n\n", "summary_of_the_review": "This paper studies an interesting research topic and connects multivariate time series forecasting with spectral GCN.. But it has the following limitations: \n1. The presentation of this paper is not clear. It is not easy to extract the main idea and intuition.  \n2.  The contribution of this paper is not clear. It is hard to tell if the author only borrows some existing conclusion in spectral GCN to time series forecasting problems or present some new finding.  \n", "strength_and_weaknesses": "Strengthes:\n1. This paper addresses the ever-changing correlations over time-series data, which is practical and complement previous works.\n2. This paper presents extensive and strong baselines in time series forecasting scenarios. Experimental results demonstrate the effectiveness of the proposed method. \n\n\nWeaknesses:\n1. The author claims this is the first work that designs a complex-valued feed-forward network in the Fourier space for efficiently computing multi-layer graph convolutions. Actually, there are some pioneer works that discuss the relation of spectral space and GNN [1,2], e.g., I wonder if the author only applies their conclusion to the multivariate time series forecasting scenarios or represents more analysis?\n2. This paper is  not easy to understand. In section 3.2, the author introduces the notation of the graph shift operator. 1) Generally, there are three types of graph shift, graph size, node feature and graph structure.  This work considers the whole three graph shift of some of it? 2) How does edge-varying connect to multivariate time series forecasting problems?   3) How does the spectral GCN address the corresponding shifted problem? Are there some motivational examples or intuitions here?\n\n\n[1] Zhu, H. and Koniusz, P., 2020, September. Simple spectral graph convolution. In the International Conference on Learning Representations.\n[2] Zhang, S., Tong, H., Xu, J. and Maciejewski, R., 2019. Graph convolutional networks: a comprehensive review. Computational Social Networks, 6(1), pp.1-23.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "1aWO9_qvOBJ", "reviewer": "Reviewer_iyiB", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper is about the combination of relations of the variables in multivariate time series (MTS) and graph neural networks for the analysis and prediction of MTS. The proposed method is to build a fully-connected supra-graph that connects variable at any two timestamps to learn high-resolution variable dependencies in an efficient way using graph neural networks. One of the contributions is the shift operator that is being proposed to reduce the computation complexity that is introduced by the supra-graph. Specifically, the authors propose to construct the Edge-Varying Fourier Graph Networks (EV-FGN) with Fourier Graph Shift Operator (FGSO) to perform graph convolutions in the frequency domain on a lower-complexity. The experimental setup includes seven datasets from different application scenarios and thirteen comparison models. The overall evaluation shows that in most cases the proposed model EV-FGN outperforms the other state of the art models. The authors include an analysis on the number of parameters and the training time, to show how the FGSO achieves an improvement in the complexity of the model. The authors also include a visualization analysis to show the patio-temporal representation that is learnt by the EV-FGN.", "summary_of_the_review": "Overall, this paper introduces a methodology that for the multivariate timeseries analysis and forecasting, using the supra graph and Fourier Graph Shift Operator to reduce the complexity of such a graph. The proposed method outperforms 13 other state of the art models when comparing to 7 datasets of various domains. The introduced methodology and concepts are hard to follow, but the authors try to introduce them, describe them and justify their decisions in a satisfactory way. However, improvements can be made to help the readers that are not familiar with these concepts (see the weaknesses section above for suggestions.).", "strength_and_weaknesses": "Strengths:\n- The paper is about an interesting topic and the authors do a good job to describe the problem and their methodology, as well as their contributions.\n- There is novelty in this work. The authors build the proposed model in existing concepts, however this is the first attempt to design a complex-valued feed-forward network in the Fourier space with a focus on reducing the complexity that the use of a supra-graph is introducing.\n- The structure of the paper is well-defined. The paper has a nice flow. The appendix also has interesting information and more details, that are useful when reading the paper.\n- The authors have done an extended evaluation of the proposed model and 13 comparison state of the art models in 7 datasets from different applications.\n- The analysis in 4.3 is nicely done and highlights the improvements and the need of the FGSO to reduce the complexity and the runtime of the proposed methodology.\n\nWeaknesses:\n- This work might have a limited interest to researchers of ICLR as the main focus is to efficiently calculate graph operations in the Fourier space.\n- The paper can be difficult to follow. The concepts presented in this work are hard to understand, especially if the reader does not have previous familiarity with them. The authors can do a better job in introducing these concepts with a few examples of how they are used in the literature. Specifically, more motivation in the introduction section is needed and specific examples of the challenges that the papers/methods that are cited in the related work section face, and how the proposed methodology will avoid such challenges. Also, a few examples in the introduction of each concept in the methodology section will help the reader understand and remember the notation and the need of each part/concept.\n- It would be interesting to see an evaluation for the multivariate time series forecasting using state of the art methods in time series representation learning as well.\n- It would also be interesting to compare the proposed methodology when not using the FGSO, how does the performance change and how does the complexity/runtime/number of parameters change?\n\n- Minor typos:\n— The word weight when used as a verb is noted as “weigh” at least two times.\n— The dataset METR-LA is noted as META-LA in Table 1. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject