Graph-based Multi-ODE Neural Networks for Spatio-Temporal Traffic Forecasting
Authors: Zibo Liu
Parshin Shojaee
Chandan K. Reddy
Venue: Accepted by TMLR ()

Abstract: There is a recent surge in the development of spatio-temporal forecasting models in the transportation domain. Long-range traffic forecasting, however,  remains a challenging task due to the intricate and extensive spatio-temporal correlations observed in traffic networks. 
Current works primarily rely on road networks with graph structures and learn representations using graph neural networks (GNNs), but this approach suffers from over-smoothing problem in deep architectures. To tackle this problem, recent methods introduced the combination of GNNs with residual connections or neural ordinary differential equations (ODE). However, current graph ODE models face two key limitations in feature extraction: (1) they lean towards global temporal patterns, overlooking local patterns that are important for unexpected events; and (2) they lack dynamic semantic edges in their architectural design. In this paper, we propose a novel architecture called Graph-based Multi-ODE Neural Networks (GRAM-ODE) which is designed with multiple connective ODE-GNN modules to learn better representations by capturing different views of complex local and global dynamic spatio-temporal dependencies. We also add some techniques like shared weights and divergence constraints into the intermediate layers of distinct ODE-GNN modules to further improve their communication towards the forecasting task. Our extensive set of experiments conducted on six real-world datasets demonstrate the superior performance of GRAM-ODE compared with state-of-the-art baselines as well as the contribution of different components to the overall performance. 

Published in Transactions on Machine Learning Research (04/2023)
Graph-based Multi-ODE Neural Networks for Spatio- Temporal Traffic Forecasting
Zibo Liu"
zbliu@ut edu marshinshojaee@nrt edu reddy@es. vt, edu
Parshin Shojaee
Chandan K. Reddy
Deportment of Computer Science, Virginia Tech, Arlington, VA
Reviewed on OpenReview: https: // openreview. net/ permformm? id= D35XXXYYV
Abstract
There is recent surge in the development. of spatio-temporal lorecasting models in the transportation domain. Long-range traffic forecasting, however, remains a challenging task due to the intricate and extensive spatio-tempora. corrclations observed in traffic networks. Current works primarily rely road networks with graph structures and learn representa- tions using graph neural networks (GNNs), but this approach suffers from over-smoothing problem in deep architectures. To tackle this problem, recent methods introduced the com- bination of GNNs with residual connections or neural ordinary differential equations (ODE). However, current graph ODE models face two key limitations in feature extraction: (1) they lean towards global temporal patterns, overlooking local patterns that are important for un- expected events; and (2) they lack dynamic semantic edges in their architectural design. In this paper, we propose novel architecture called Graph- based Multi-ODE Neural Networks (GRAM-ODE) which is designed with multiple connective ODE-GNN modules to learn bet- ter representations by capturing different views of complex local and global dynamic spatio- temporal dependencies. We also add some techniques like shared weights and divergence constraints into the intermediate layers of distinct ODE-GNN modules to further improve their communication towards the forocasting task. Our extensive set of experiments con- ducted on six real-world datasets remenstrate the superior performance of GRAM-ODE com- pared with state-of saate-of-hhe-att baselines as well the contribution of different components to the overall performance. The code is available at https:///ittheii comptttoooooooooeeeeoooooocoetet wtwpa/www.ow.ww////eeeeeeeeeeeee 2tli://////////000
1 Introduction
peatooltemporal forecasting is one of the main research topics studied in the context of temporally varying spatial data which is commonly seen in many real-world applications such traffic networks, climate net- works, urban systems, etc. (Jiang & Luo 2022 Du et al. 2017 Jones 2017 Longo et al. 2017). In this paper, we investigate the problem of traffic forecasting, in which the goal is to statistically model and iden- tify historical traffic patterns in conjunction with the underlying road networks to predict the future traffic flow. This task challenging primarily due to the intricate and extensive seatio-temporal dependencies in traffic networks, also known as intra-dependencies (i.e., temporal correlations within one traffic series) and inter-dependencies (i.e., spatial correlations among correlated traffic series). In addition to this, frequent events such traffic peaks accidents lead to the formation of non-stationary time-series among different locations, thus, posing challenges for the prediction task.
Traffic forecasting is spatio-temporal prediction task that exploits both the location and event data collected by sensors. Traditional methods such as AutoRegressive Integrated Moving Average (ARIMA) and Support Vector Machine (SVM) algorithms only consider the temporal patterns and ignore the corresponding spatial relations [Jeong et al. 2013 Williams & Hoel 2003
Van Der Voort al. 1996). Statistical and classical
* "Corresponding author

---

Published in Transactions on Machine Learning Research (04/2023)
Prior Models
Our Model
Node
Local Correlation
Node-Edge Correlation
Edge Global Correlation
Edge-Edge Correlation
An overview of the proposed model alongside with prior models. The sub-figures depict traffic dala over time with blue nodes representing recording points and lines representing roads. The orange dashed line shows the common global temporal pattern. Our model incorporates local temporal patterns represented by the purple dashed line, node-edge correlations depicted by red arrows, and dynamic edge-edge correlations displayed by green ares.
Figure 1:
machine learning methods suffer from limitations in learning complex spatio-temporal interactions, and thus deep learning models were later introduced for this task. Early examples of deep learning approaches to capture spatial and temporal relations are FC-LSTM (Shi et al. 2015) where authors integrate CNN and LSTM modules; and ST-ResNet (Zhang et al. 2017 which uses deep residual CNNs for both spatial and temporal views. However, these CNN-based methods are developed lowards grid data and cannnot account for traffic road networks which are more akin to graph-structures. Hence, researchers in the domain have recently developed Graph Neural Network (GNN) based approaches for effectively learning graph-basce representations. Examples of these graph-based models are: STGCN (Yu et al.
2018 which utilizes complete convolutional structures for both temporal and spatial views; and DCRNN (Li et al. 2018b) which combines the diffusion process with bi-directional random walks on directed graphs in order to capture spatio-temporal dependencics.
However, such GNN-based methods cannot capture long-range spatio-temporal relations or develop deeper representations due to their limitation of over-smoothing (Lan et al. 2022 Li et al. 2018a Zhou et al. 2020). Deep GNN over-smoothing occurs when a GNN model with deeper architecture tends to lose the discriminative ability and learn similar node representations for all nodes. Thus, making it challenging learn richer representations and investigate more complex graph structures. In order to address this problem, rescarchers have introduced combining GNNs with residual skip connections (Chen et al.) 2020 which are connections that. bypass one or more layers and allow information to flow more Treely through the network, therefore, improving the network's ability in learning more complex temporal relations. Neural Ordinary Differential Equation (NODE) is a type of deep learning model that uses continuous-time dynamics to learn more powerful representations of the time series data. NODEs can also be used to address over-smoothing in GNNs by providing a more flexible and expressive model architecture capturing temporal relations. Recently, this was studied by the STGODE (Fang et al., 2021 model that integrates GNNs with NODEs CCen et al. 2018) to model the dynamics of traffic over time. This combination can derive continuous GNN (CGNN) with continuous temporal connections toward alleviating the werrssoothing problem. Nevertheless, current CGNN models still encounter the following limitations: (1) Previous approaches in this area tend to overemphasize the global lemporal structures while undervaluing local patterns, which are often crucial for predictions in the presence of unexpected traffic events, e.g., a road has higher chance of getting clogged shortly after a car accident. This will cause drivers switch this road may significantly drop for short period of time. Therefore, ignoring these local temporal patterns may cause significant problems in the final predictions for roads facing unexpected events. (2) The dynamic
faster route and thus the traffic flow on
2

---

Published in Transactions on Machine Learning Research (04/2023)
correlation of traffic nodes is ignored by existing approaches. In other words, models usually do not consider the dynamic semantic spatial correlations (i.e,, dynamic semantic edges) in their architecture. (3) Several baselines use vanilla aggregations, such as average pooling, to combine latent features learned from multiple streams which ignore the high-dimensional feature interactions.
To overcome the above limitations, we propose a novel framework called GRAM-ODE, GRAph-based Multi- ODE Neural Networks. First, in order to balance the consideration of global and local temporal patterns, we design new ODE modules for the local temporal patterns in addition to the existing ODE module for the global temporal patterns (Fang et al., 2021) using different sizes of temporal kernels (represented purple and orange dashed lines in Fig. 1. Specifically, for local dependencies, we assign ODE functions to the local kernel's output that approximate local patterns, and then concatenate the results. These local and global ODE modules are depicted with more details in Fig. 3(a) and Fig. 8 B(),, respectively. Second, we design a new ODE module into our model to consider the dynamic correlation of traffic nodes as well as edges. In other words, at each time step, we find the intermediate dynamic spatial correlations basod on the embedding represenlations of nodes (i.e., dynamic semantic edges), and then construct new ODE module to approximate patterns of semantic edge-to-edge correlations over time (represented with different odge patterns over time in Fig. 1. More details rogarding this dynamic semantic edge ODE module are provided in Fig. B(c). Third, we also design the nonlinear aggregation paradigm and a multi-head attention across different ODE modules (represented in Fig. [(d) and different streams of traffic graphs (represented in the last layer of Fig. 2), respectively. Through these two operations, we account for high-dimensional correlations and similarities of latent features corresponding to different ODE modules and traffic graphs. By doing so, let the model select and fuse latent features from different. views for the forecasting task.
Also, since our proposed GRAM-ODE includes multiple ODE modules, we ensure that these different modules are not isolated and have effective connectivity in the intermediate layers. To do so, we design coupled ODE modules in two ways: (1) adding similarity constraint between the semantic parts of local and global modules to ensure those two semantic embeddings do not diverge from one another (represented with red marks in Fig. 3); (2) sharing weights for the global node-based and edge- based ODE modules (represented with green marks in Fig. 3). Therefore, we create coupled graph-basen based multi-ODE structure as GRAM-ODE in which all modules are designed to effectively connect with each other for the downstream application of traffic prediction. The major contributions of this work are summarized below.
Developing new ODE module for capluring local lemporal pallerns. Due to the importance of short- term temporal dependencies in the traffic prediction in case of unexpected events, develop new ODE module for short-term dependencics in addition to the current ODE block for global temporal patterns.
Developing new ODE module for the dynamic semantic edges. In addition ODE blocks for traffic nodes, which model the dynamic node-to-node correlations over time, we also add a new ODE module for the dynamic semantic odges based node reprosentations (i.e., dynamic spatial correlations) to model the edge-toeedge correlations over time.
Building effective connections between different. ODE modules (coupled multi-ODE blocks). To build effective intcractions between multiple ODE modules, we consider shared weights for node-based and edge-based ODE modules as well as adaptive similarity coustraints for the outputs of local and global temporal ODE modules.
Designing a new aggregation module with multi-head attention across features of different streams. To enhance the model's awareness in the selection and fusion of different ODE modules as well as the streams corresponding to different types of traffic graphs, we design multi-head attention mechanism the aggregation layers.
The rest of this paper is organized as follows. Scction 2 summarizes existing traffic forccasting works based on machine learning, graph-based and NODE-based hods. Section 3 provides some of the required pre- liminaries and explains the problem formulation. Section A covers the details of our proposed GRAM-ODE method and its different components. Our experimental evaluation including both quantitative and quali- tative comparison results, ablation studies, and robustness assessments are reported in Section 3 Finally, Section 0 concludes the paper.
3

---

Published in Transactions on Machine Learning Research (04/2023)
2 Related Works
2.1 Machine Learning and Deep Learning Methods
Researchers have employed traditional statistical and machine learning methods for the task of traffic fore-
casting. Some prominent example models are (1) K-Nearest Neighbor (KNN) (Zhang et al. 2013) which
predict traffic of a node based on its k-nearest neighbors; (2) ARIMA (Van Der Voort et al. 1996 Alghamdi
ct al.] 2019) which integrates the antoregressive model with moving average operation: and (3) SARIMA
(Williams & Hoel 2003) which adds a specific ability to ARIMA for the recognition of seasonal patterns.
Many of these machine learning models only consider the temporal dependencies and ignore the spatial information. Also, they are usually based on human-designed features and have limitations in learning infor- mative features for the intended task. Later, deep learning methods became popular due to their ability in considering the complex and high-dimensional spatio-temporal dependencies through richer representations. Early examples of deep learning models considering the spatial and temporal relations are FC-LSTM (Shi et al. 2015), COnVLSTM (Xingjian et al.) 2015), ST-ResNet (Zhang el al.] 2017), ST-LSTM (Liu et al. 2016), and STCNN (He al.) 2019) which are usually based on convolutional neural networks (CNNs) and recurrent neural networks (RNNs) to account for spatial and temporal information. However, these models are developed for the grid-based data and disregard the graph structure of traflic road networks. Due to this limitation, researchers moved into the application of graph- based deep learning models like graph neural networks.
2.2 Graph-based Methods
Graphs provide vital knowledge about the spatial, temporal, and spatio-temporal relationships that can potentially improve performance of the final model. Recently, researchers employed the graph convolu- tion network (GCN) (Kipf & Welling 2017) to model the spatio-temporal interactions. DCRNN (Voeriah) et al. 2015 is an early example of it that utilizes diffusion GCN with bi-directional random walks to model the spatial correlations as well as a GRU (Gated Recurrent Unit) based network to model the temporal correlations. GRU is an RNN--ased method which is not effective and efficient in modeling long-range temporal dependencies. To address this limitation, works such STGCN (Yu et al., 2018) and Graph- WaveNet (Wu et al., 2021) utilize convolutional operations for both spatial and temporal domains. After the rise of attention- based models in deep learning, researchers realized that these two models still have some limitations in learning spatial and temporal correlations due to the limited capability of convolutional operations in capturing high-dimennsonal correlations. Therefore, two attention layers are later employed in ASTGCN (Guo et al.] (219)) to capture the spatial and temporal correlations. However, these models are limited in capturing local relations and may lose local information due the sensitivity of representations to the dilation operation. STSGCN (Song et al.) 2020) used localized tpatio-temppral subgraphs to enhance prior models in terms of capturing the local correlations. However, since the model formulation does not incorporate global information, this model still had limitations when it came to long-term forecasting and dealing with data that included missing entries or noise. In addition to the spatial graphs from predefined road networks, STEGNN Li & Zhu, 2021) later introduced the use of Dynamic Time Warping (DTW) for data-driven spatial networks which helped the model to learn representations from different data-driven and domain-driven views. STFGNN also utilized a new fusion module to capture spatial and temporal graphs in parallel. However, due to the over-smoothing problem of deep GNNs (which happens when a GNN model learns similar node representations for all nodes in case of adopting deeper architecture), current GaN-based models incur some difficultics in learning rich spatio-temporal representations with deep layers.
2.3 Neural Differential Equation Methods
Neural Differential Equations (NDEs) (Kdger 2022 provide new perspective of optimizing neural networks in continuous manner using differential equations (DEs). In other words, DEs help to create continuous depth generation mechanism for neural networks, provide a high-capacity function approximation, and offer strong priors on model space. There are a few types of NDEs: (1) neural ordinary differential equation
(NODE) such as basebased RNN models [Habiba & Pearlmutter 2020 Lechner & Hasani 2022) (e.g., ODE-RNN, ODE-GRU, ODE-LSTM) and latent ODE models (Rubanova et al.] 2019); (2) neural controlled differential equation (NCDE) (Choi et al. 2022 Kidger al. [2020] Morrill et al. (021)) which is usually used
to learn functions for the analysis of irregular time-series data; and (3) neural stochastic differential equation
4

---

Published in Transactions on Machine Learning Research (04/2023)
Table 1: Notations used in this paper.
Notation
Description
G
Traffic graph
Set nodes
E
Set edges
AC
Connection adjacency matrix
18E
adjacone matrix
D
Degree matrix
Normalized adjacency matrix
fe/ Jel ft
Message generation for Edge/Global/
/Global/ /Local Neural ODEs
ODE function (iutegrals from é) initialized with R(U)) Hrlge/Glohal) Local ODE function features Historical time series data
(4) HH (1)/(()
Future time scries data
Y
Predicted future time series Length historical Length of target time series Number nodrss [V] Number chantels Input of multi ODE-GNN
L
series
E'
N
И
A
Updated adjarrncy matrix with shared spatial weights
T. TT
Updated Global/Local /Loaa///ge//// temporal shared weights
EM
Edge message
GM
Global message
LM
Local
RadC)
Features of single embedded time step in Local ODE-GNN
E"
Length of the embedded temporal window in Local ODE-GNN Local ODE-GNN's output the embedded temporal step Learnable threshold parameter in message filtering Embedddings of n-th ODE module in aggregation layer Output multi DDE-s aggregation layer
K(()
e
W. W./be
Weight matrix/biin /ias vecsor in update layer Output multi ODE-GNN's update layer Hidden states of (-th TCN's layer
H"
Embedding size of L-th TCN's layer
W
Convolutional kernel of 1-th TCN's layer
Exponential dilation rate of [-th TCN's layer
W.
attention query /key vaalue wright The attention query/key /key /value blas vector Number atleation heads
/ bef De
Attention output the 4-th head
Erzor threshold in the Huber loss
2021b a) which usually employed for generative models that can represent complex stochastic dynamics. More recently, NODEs have been employed for the traffic forecasting task (Fang al.,
(NSDE)
Kidger et al.
2021 Su al. 2022 Pu al. 2022 For example, STGODE (Fang et al., 2021 is proposed to address the aforementioned over-smoothing problem of GNN-based models. STGODE provides new perspective of continuous depth generation by employing Neural Graph ODEs to capture the spatio-temporal dependencies. However, complex apatio-temporal correlations such as the lack of local temporal dependencies and dynamic node-edge communications are not captured by this model. In this study, we employ the idea of Neural Graph ODE for multiple coupled ODE-GNN modules which take into account the local temporal patterns and dynamic spatio-lemporal dependencies, and as a result, can provide belter function approximation for the forecasting in the presence of complex apatio-tempora correlations.
3 Problem Formulation
This section explains the required preliminarics and definitions for GRAM-ODE, and then defines the main problem statement. All notations used this paper are sunmarized in Table [
3.1
Definitions
Definition Traffic Graphs. We consider the traffic network graph = E, where V is the of nodes, E is the set of edges and A the adjacency matrix such that A NxN if [V| N. In this paper, types of graphs, the connection map graph and the DTW (dynamic time warping) graph. In the connection map graph, the adjacency matrix, denoted by A represents the roads and actual connectivity among traffic nodes with binary value.
A =
1, if and vy are neighbors 0, otherwise
(1)
5

---

Published in Transactions on Machine Learning Research (04/2023)
where v and by refer to traffic nodes i and in the graph. In the DTW graph, the adjacency matrix. indicated by ASE, is generated from the Dynamic Time Warping (DTW) algorithm (Berndt & Clifford 1994) which calculates the distance between two time-series corresponding to a pair of nodes.
1, DTW (X) <E
A
(2)
0, otherwise
where X and X refers to the time-series data of nodes and J. respectively; also identifies the sparsity ratio of adjacency matrix. Notably, DTW is more effective than other point-wise similarity metries (such Euclidean distance) due to its sensitivity to shape and pattern similarities.
Graph Normalization. Adjacency matrix A E {AcC (CCA E N is normalized with D- AD- where D is the degree matrix of A. As shown in Eq. (3), the self-lopp identity matrix I is incorporated in normalization to avoid the negative eigenvalues.
Definition 2:
A= ( +AD
(3)
where Å is the normalized adjacency matrix; and a E 10 identifies the eigenvalue range to be in a.).
Definition 3: Graph-based Neural ODE The standard formulation of GNN-based continuous-time ODE function is defined in Eq. E It takes the initial value H(0), temporal integrals from to given time t, traflic graph G, and the Neural ODE's network parameter 0.
H(t) = H(0) + L
(4)
where f is the process to generate semantic message of Neural ODE parameterized by 0 to model the hidden dynamics of graph G. Since the structure of our proposed method consists of multiple ODE-GNN blocks, we have different types of Eq. 44 graph neural ODEs which are explained with more details in Section $
Definition 4: Tensor n-mode multiplication. We use subscript to identify the consor-matrix multiplication on the specific dimension as shown below.
(B x C)al=
Bi
(5)
where B E R N. CE Nox,, so, 822 C C.RNNNM The n-mode tensor-matrix multiplication is denoted as Xn with the nith subscript.
3.2 Problem Statement
The spatio-temporal forecasting problem is described as learning a mapping function F that transforms the current historical aratio-temporal data X' = XI-
into the future spatio-temporee data y (X.+1, X142, X(+tr), where and L' denote the length of historical and target time-scrics to be predicted, respectively. In the traffic forecasting problem, we have historical tensor X' E Bx and traffic graph 9, where B is the batch size; N is the number of nodes; T. is the input temporal length; and C is the number of input features (e.g., traffic speed, flow, density). The goal is to find ÿ = F(X, f. in which F is the overall forecasting network and f corresponds to different graph- based Neural ODE processes.
4
GRAM-ODE
4.1
Overview
The overview of our proposed GRAM-ODE is shown in Fig. 2 which is composed of two streams of operations: (i) DTW-based graph (top row) which is constructed based on semantical similaritics, and (ti) connection map graph (bottom row) which is constructed based on geographical spatial connectivities. These two types of adjacency matrices are fed into the model separately to capture spatial correlations from both data-driven and geographical domain-driven views. They are later integrated with the multi-head attention mechanism in the final layer. As shown in Fig. 2 we have three parallel channels for each graph, each of which has two consecutive GRAM-ODE layers with multi ODE-GNN block sandwiched between two blocks of temporal

---

Published in Transactions on Machine Learning Research (04/2023)
The process to build the graph
TCN
AM: Attention Module
AM
repeat twice
Muiti ODE-GNN
Figure A graphical overview of the proposed GRAM-ODE framework. The DT'W-based graph will be obtained from data at the blue triangle mark. The connection map will be obtained from the distance among recording nodes at the blue diamond mark. These two traffic graphs are separately imported into the model which consists of three parallel channels of two consecutive GRAM-ODE layers. Each layer contains sandwiched multi ODE-GNN block (explained Fig. Bb between temporal convolution networks (TCNs). Outputs of these streams are then aggregated with an Attention Module (AM) in the final layer.
dilated convolution (TCN). The final features of each graph from different channels are then concatenated and imported into the attention module (AM) which is designed to effectively fuse these two scparate sets of operations by laking into account all the high-dimensional relations towards the intended forecasting task. We provide further details about each of these components in the subsections below.
4.2 GRAM-ODE Layer
Each GRAM-ODE layer consists a multi ODE-GNN block placed between two TCNs. Fig. 3 illustrates details of our proposed Multi ODE-GNN block in which we combine multiple ODE modules take into account all the essential temporal and spatial patterns from different viewpoints and extract informative features from their fusion. In the multi ODE-GNN block, we design three types of message passing operations, message filtering constraint as well as new aggregation module to consider high-dimensional correlations.
4.2.1 Message Passing Layer
The current models primarily focus on node-based global temporal correlations and overlook short-term temporal patterns as well as the dynamic edge-based correlations in their formulation. Hence, we design three types of message passing processes to formulate ODE modules corresponding to global, local, and edge-based temporal dependencies.
Shared Weights: Although node-based and edge-based features have distinct semantic meanings, they can still exhibit common spatial and temporal patterns. In order to consider these common patterns in problem formulation, design shared weight matrix between node- based and edge-based global temporal ODE modules (shown in Fig. a(a) and B(c).. We define two weight matrices for consideration of these shared spatial and temporal weights. The shared spatial weight, denoted by M, is added to the normalized adjacency matrix as Á == A + M, where M is initialized randomly from the normal distribution. The shared temporal weights, denoted by W.1, W.2, added to the node-basod and odge-basod global temporal modules, given Eqs. ( and 7 To consider the local temporal patterns model formulation, we apply different sizes of temporal kernels Wn, Wia into the local temporal module shown in Eq. 88.
7

---

Published in Transactions on Machine Learning Research (04/2023)
Mean
H,(D)
H2(t)
Linear
Linear
EM
Copy times
(b)
Hy
(t)(t)
H
Linear
Linear
AM
LM
Shared Weights A&T
ODEs
(,
H
GM
Linear
Constraint
Lower Limit to Integral
Upper Limit to Integral
Sigmoid
Figure 3: An overview of the multi ODE-GNN block which consists of three ODE modules, i.e., (a) global, (b) local, and (c) edge-based temporal dependencies as well (d) new aggregation layer. The imputs and outputs of the multi ODE-GNN block are displayed with and H' blocks on the left and right sides of the diagram. The shared weights among different ODE modules marked in green, and constraint to limit the divergence of embeddings is marked red. AM denotes the Attention Module defined in Section 4.3
Te = (He(t) x Wa) (Ha(tt) X4 Ws2) T (HG (t) X3 (a)) (t) X3 W2
(6)
(7)
T = (Hl (t) X3 (H) (t) X3 Wi2)
(8)
where Ws1, Ws2 R L.xt represent the global temporal shared weights with referring the global temporal length; Wn, Wa RIx1 represent the local temporal weights for each time step in the embedded temporal space; and (t),), (t), and Ht( represent the feature of edge, global, and local ODE module, respectively.
Global and Local Message Passing: Fig ((a) represents the global message passing with the goal of modeling long-term mode-based temporal patterns in continuous manner. Eqs. (9) and (10) represent the global message processing operations and return the global message GM, respectively. Additionally, Fig 3 (b) represents the local message passing operations on local temporal data to emphasize the importance of short-term temporal patterns. this module, we first use self-attention mechanism with dense projection layers to create the input in the lower-dimensional embedded temporal space by considering all the possible high-dimenssonal correlations (similar to the attention module (AM) explained with more details in Section 4.3 Then, features of each time stamp in the embedded temporal inputs are separately imported into the local ODE functions, encouraging the network to learn implicit local dependencies. These features at each embedded time step are denoted by HA E BBxNXIXC whore i E {0, 1, 2, 3..,, L" -1], and L" is the size of the embedded temporal window. As shown in Eqs. 13 and 15 at each embedded time step, HAs used as the initial value formulate temporal dependencies of futurre & I./T." time stamps which are returned as K(i). Then, the outputs are concatenated to form the final local message LM [<()]]/[(]]]]] ... [[(L"-
H₂(0) J. (,HH,(T)) wjdr fg = (t) x2 (19))) (t) X4 (W II
GM
(9)
(10)

---

Published in Transactions on Machine Learning Research (04/2023)
LM hww.=oooooMesssggesssssssggggo/ H4(0), A40 [ HAu.HAI HAU ATT(H((0))
(11)
(12)
X(1) -iiHAAi (o) (o))l (B
(13)
Fi(H. t2) =.Ai + {²¹ (.(7
(14)
fi fi=Hi:( X2 (A- ((S(T)) -((((()))))((())))))) HAilt) X4 (W- -
(15)
c depicts the edge message passing procedures that take into account the dynamic edge-based temporal patterns in addition to the node-based correlations. We first create the initial edge features (0) E fx xx N. from the node representation H E Bxxxxxxx by taking an average over represent the edge message passing operations
Edge Message Passing: Fig.
C channels and copying the N dimension. Eqs. 16 and 17 which return edge message EMM
-o(00) + / A,(HL(T),,,TT)dt To)dr
EM
(16)
f. = (t(t X2 (Å 1) + (c(()((((T))
(17)
In Eqs 00 17 Hg: Hi, and He represent the feature of global, local, and edge ODE modules, repsectively; Ty Tr, and Te represent the global, local, and edge temporal weights formulated in Eqs. 66 respectively; Á represents the updated adjacency matrix based on shared spatial weights; W E R cCxC represents the weight matrix for modeling interaction of different channels; S(.) shows the sigmoid operation; and the identity matrix.
4.2.2 Message Filtering
To prevent the semantic parts of local and global modules diverging from one another, add message filtering constraint into the problem formulation (shown with red mark in Fig. 3) Eq. 118) shows that
this message filtering constraint works like a both-way clipping operation in which we clip local semantic embeddings they are significantly larger or smaller than global semantic embeddings.
GM + tc, ef if LM > GM +e
LM =
GM - e, if LM < GM - LM, otherwise
(18)
where LM and GM represent the local and global semantic embeddings; and represents learnable noise parameter initialized with normal distribution.
4.2.3 Aggregation Layer
Fig. 3(d) represents our aggregation paradigm designed for combining output features of the three ODE modules. Instead of employing single add or pooling operations in the aggregation layer, we use the non- linear matrix multiplication operation account for key correlations in the higher dimensions. Eq. (19)
represents our designed aggregation in which the output of each ODE module multiplied by the sum of other modules that are normalized by softmax operation. This gated aggregation has several benefits: (i) it enables the selection of features that are more crucial for forecasting; (ti) it allows for non-linear aggregation; and (iii) it contains linear gradient path, reducing the risk of vanishing gradient.
H' Aggregation EM)
Σ Σ Pm softmaz(p.)
(19)
2K
nmm
where H' represents the output of Multi ODE-GNN aggregation layer; LM, GM, and EM represent the local, global, and adge-based semantic embeddings, respectively, K = refers to three ODE modules (pu GM, P1 LMM, p2 FM)); and operation represents the point-wise multiplication.

---

Published in Transactions on Machine Learning Research (04/2023)
4.2.4 Update Layer
After obtaining the aggregated information, we add residual connection around the multi ODE-GNN block to update the node information of GNN. To achieve this, the imput of multi ODE-GNN block is remapped to the aggregated output using Fully Connected Layer as shown below.
H" = UUpdate((H .))
Sigmoid(W, + b,) + B H'
(20)
where H and H' represent the imputs and outputs of the multi ODE-GNN block, respectively; W,, and by denote the weight matrix and bias vector of the residual fully connected layer, respectively. Also, a and B are the hyperparameters identifying the combination weights in the residual connection.
4.2.5 Temporal Convolutional Network
Since the problem being considered in this paper is spatio-temporal in nature, we need to consider the temporal correlations of nodes in addition to their spatial relations in the problem formulation. To model the temporal correlations, we utilize temporal convolutional networks (TCNs) which usually have more efficient training, more stable gradients, and faster response to dynamic changes compared to recurrent neural networks (RNNs). The architecture of TCN adopts the dilated convolutional operations with 1-D kernels along the time axis.
fx H = Sigmoid (Wi C d' d(H)) 1= 1=1,2....,
1=0
(21)
where X E BBXNXLXC is the imput of TCN; Htm BBXNxLXCC is the latent output with C' the embedding size in the 1-th TCN's layer; W represents the convolutional kernel of the I-th layer; and d' is the exponential dilation rale which is used to expand the receptive field during the convolution. We usually take d' 2/-1 in the temporal dilated convolution.
Algorithm D provides the pscudocode for the GRAM-ODE layer by sequentially passing the input via TCN, Multi ODE GNN, and another TCN blocks. The previously explained steps of Multi ODE-GNN block including initialization, message passing, message filtering, aggregation, and update are summarized in lines
P6
Algorithm 1: GRAM-ODE Layer
Input: Node Information X, Traffic Graph g
Output: Updated Node Information X
1 # TCN Block
TCN(())
3 # Find Initial Values
H. (0) + Н
H.::(G) +- ATT(())
A < Mean(H) # average chanuel dimension
H. (0) Repeat(Af # repeat N times
8 # Edge Message Passing
9 EM + (t)(t) Xa (A 1A-11 1)+ HL,())(((T)) - I))
10 # Global Message Passing
GM + (1) X2 (Å (Ā-1) I) ((S(T)) 1114 (0)) (0,(0) X4 (W-
12 # Local Message Passing
LM (Ai(t) (t) (A- 1) + ((S(T)- ())) -aallt) X4 (W -1
# Message Filtering
15 If LM > GM+e then
16 else if I.M < GM-e then
LM + GM +
LM GM
19
20
LM LM
end
Aggregate Multi ODE-GNNs
29 P1. + GM,, LM, EM
K K
24 H' IR Σ Σ
O softmaz(p..)
Update
H" &- Sigmoid bx) + B H
27 TCN Block
28 X TCN(H")
10

---

Published in Transactions on Machine Learning Research (04/2023)
4.3 Attention Module
We use the attention mechanism in the last layer of GRAM-ODE effectively aggregate the final learned embeddings of two traffic graphs (i.e., DTW-based graph and the connection map graph) in way that is better aligned with the forecasting objective. The attention module (AM) is designed to replace the previous fully connected layers while capluring the correlations of high-dinensional information. In this module, we first concatenate the embeddings of two graphs and then compute the attention scores between them. Eqs. 22 and [23 mathematically describe this attention operation.
Q XW + by: K =XW + bk V= =X + +b,
(22)
X soflmar (Vo- (QTK.) V
(23)
where Wg (by),,, Wx(bs), and (b.)))) represent the altention query, key, and value weight matrix (bias vector), h respectively; X the input of allention module; the number of heads; and v T is the normalization factor with C' representing the embedding dimension. Therefore, Q: ...., Q+), K:{K1, K2, ..., Ka);
V:[V, V., } shows the query, key, and value set for multiple atlention heads. Finally, output of the attention module X can be mapped to the feature space of original data for prediction with linear dense layer.
4.4 Loss Function
In regression problem settings (such as the traffic flow forecasting), Huber loss function between the real value and predicted value is widely used in the literature. It combines the merits of L1 and L2 loss functions. Huber loss is a piece-wise function which consists of two parts: (1) squared term with small and smooth gradient when the difference between the true and predicted values is small (i.c., less than a 5 threshold value) and (2) a restricted gradient lerm when the true and predicted values are far from each other. Eq. 24 represents the standard form of Huber loss function.
2 y));
))
(24)
81 3 1 00 otherwise
where 8 is hyperparameter value set for the intended threshold; y is the true future spatio-temporal data; and y the predicted future data.
Algorithm 2 provides the complete pseudocode of GRAM-ODE training. In each optimization step of this algorithm, the outputs of all GRAM-ODE layers across parallel channels are concatenated, and these con- catenated outputs of different graph types are then fed into the attention modulc for better aggregation and prediction.
Algorithm 2: GRAM-ODE training
Input: Historical Data Future Data V, Traffic Craph
Output: Forecast Model with parameter
initialize model parameters
normalize the historical data X t- X-monx(X)
sta(X")
3 for number of epochs until convergence do
for batch num batches do
5 layers=[]
6
for graph g in G do repeat
X
FRRM--DE Layer(X.g)
layers (layers, (X)) # Concatenation
until nump_prralleeed layers
end
X Attention Module(layers)
18
í - X' x std(2') + mean(X))
{< Huber Loss(), # Compute Loss
8 + f - Ael # Update Parameters
16
end
17
11

---

Published in Transactions on Machine Learning Research (04/2023)
5 Our Experiments
We conduct experiments on six real-world datasets and seven baseline models to evaluate the effectiveness of our proposed GRAM-ODE and its components for the traffic forecasting task.
5.1 Datasets
We show the performance results of our model on six widely used public benchmark traffic datasots PEMS03, PEMS04, PEMS07, and PEMS08 released by (Song et al. 2020 as well as PEMS-BAY BAY (Li et 2017) and METR-LA (Jagadish et. al.) 2014h. The first four datasets (PEMS03, PEMS04, PEMS07, PEMS08) are collected based on the California Districts they represent (District 3, District 4, District 7, and District 8, respectively). PEMS-BAY covers the San Francisco Bay Arca and METR-LA focuses on the traffic data of the Los Angeles Metropolitan Area. All these datasets collect three features (flow, occupation, and speed) at each location point over a period of time (with 5-minute time intervals). The spatial connection network for each dataset constructed using the existing road network. The details of data statistics are shown in Table.
To use these datasets in experiments, we pre-process the features by z-score normalization.
Table 2: Basic statistics of the datasets used in our experiments.
Data
PEMS03
PEMS04
PEMS07 PEMSOS
PEMS-BAY METR-LA
Location
CA. USA
Time Span
9/1/2018
1/1/2018
5/1/2017
7/1/2016
1/1/2017 5/31/2017
3/1/2012
11/30/2018
2/28/2018
8/31/2017 8/31/2016
3/30/2012
Time Interval
5 min
Sensors
358
307
207
Edges
547
340
866
295
2,369
1,515
Time Stops
20,208
16,992
25,224
17,856
52,116
31,272
5.2
Evaluation Metrics
We use Mean Absolute Error (MAE), Mean Absolute Percentage Error (MAPE), and Root Mean Squared Error (RMSE) metrics to evaluate the spatio-temporal forccasting. These motries are defined as follows.
MAE
n M 0el. MAPE
12 1 (- Σ 11
y/1 DE 1) 100%,
RMSE
Σ 11
1 n
01
Y
Vi
5.3
Baselines
We compared our proposed GRAM-ODE with the following basclines.
ARIMA [ox & Pierce 1970 ; Auto-Regressive Integrated Moving Average is one of the most well-known
statistical models for time-series analysis.
DCRNN (Veeriah et al. 2015:: Diffusion Convolutional Recurrent Neural Network utilizes diffusion graph convolutional networks with bidirectional random walks on directed graphs, and seq2seq gated recurrent unit (GRU) to capture spatial and temporal dependencies, respectively.
STGCN Yan al. 2018): Spatio-Tempont Graph Convolutional Network combines graph structure convolutions with ID temporal convolutional kernels to capture spatial dependencies and temporal cor- relations, respectively.
GraphWaveNet (Wu et al. 2021 :: GraphWaveNet integrates adaptive graph convolution with 1D dilated
casual convolution capture spatio-temporal dependencies.
STSGCN (ong et al. 2020 Temporal Synchronous Graph Convolutional Networks decompose the problem into multiple localized paaio-tempora subgraphs, assisting the network in better capturing of spatio-temporal local correlations and consideration of various heterogeneities spatio-tempora. data.
Datasets are downloaded from STSGCN github repository httpp:://gttuub.com//aaiidaaaai
12

---

Published in Transactions on Machine Learning Research (04/2023)
Table Performance comparison of GRAM-ODE and baselines on six benchmark datasets. A lower MAE/MAPE/RME indicates better performance. The best results are in bold and the second-best are under- lined.
Dataset
Metric
ARIMA
DORNN
STGON
GraphWaveNet STSGCN
STEGNN
STGODE
GRAM-ODE
MAE
33.51
18.18
17.48
19.85
17.48
16.77
16.50
15.72
PEMSO3
MAPE(%)
33.78
18.91
17.15
19.31
16.78
16.10
16.69
15.98
RMSE
47.59
30.12
29.21
28.94
87.84
MAE
33.73
21.70
22.70
25.45
21.19
20.81
20.81
19.55
PEMS01
MAPE(%)
24.18
17.12
14.59
17.29
13.90
13.02
13.77
12.66
RMSE
48.80
38.12
35.55
39.70
33.65
32 12.51
32.
31.05
MAE
38.17
25.30
25.38
26.85
24.26
23.
22, 99
21.75
PEMSO7
MAPE(S)
19.46
11.16
11.08
12.12
10.21
9.21
10.14
9.74
RMSE
59.27
38.58
38.78
42,78
39.03
36.60
37.54
34.42
MAE
31.09
17.86
18.02
19.13
17.13
16.94
16.81
16.05
PEMSOS
MAPE(%)
22.73
11.45
11.40
12.68
10.96
10.60
10.62
10.58
RMSE
44.32
27.83
27.83
31.05
26.80
26.22
25.97
25.17
MAE
3.38
2.07
2.49
1.95
2.11
2.02
2.30
1.67
PEMS-BAY
MAPE(%)
8.30
5.79
4.96
4.79
3.83
RMSE
6.50
4.74
5.69
4.48
4.85
4.63
4.89
3.34
MAE
6,90
3.60
1.59
3.53
3.65
3.55
3.75
3.41
METR-LA
MAPE(%)
17.40
10.50
12.70
10.01
10.67
10.56
10.26
9.38
RMSE
13:23
7.60
9.40
7.37
7.81
7.47
7.37
6.64
Spatio- Temporal Fusion Graph Neural Networks uses Dynamic Time Warping (DTW) algorithm to gain features, and follow STSGCN (Song et al. 2020 in using sliding window to capture spatial, temporal, and spatio-temporal dependencies.
STEGNN (L & Zhu
2021):
STGODE (Fang et al. 2021 Opatio-Tempont Graph ODE Networks attempt to bridge continuous differential equations to the node representations of road networks in the area of traffic forecasting.
5.4 Experimental Settings
Following the previous works in this domain, we perform experiments by splitting the entire dataset into 6:2:2 for train, validation, and test sets. This split follows temporal order, using the first 60% of the time length for training, and the subsequent 20% each for validation and testing. We use the past one hour to predict the future one hour. Since the time interval of data collection is 5 minutes, L, I,' = 12 temporal data points. Based on Table 2 we have different number of sensors among different datasets, therefore, |V| is different. DTW threshold (c) in Eq. (2) is 0.1; number of channels (C) in the historical data is 3 (i.e., flow, speed, and occupation) and in the embedding space is 64. The shared temporal weights We1, Wa2 E IR 12x12 are initialized randomly from normal distribution. The length of latent space for the input of local ODE block is L" 4, and in the final allention module, number of altention heads h 12. During training, we the learning rate of 10 10-4, 10-4, 10-5, 10-5, 10-4, and 10-4 for PEMS03, PEMS04, PEMS07, PEMS08, PEMS- BAY, and METR-LA datascts, respectively. The optimizer is AdamW. All experiments are implemented using PyTorch Paszke et al. 2019) and trained using Quadro RTX 8000 GPU, with 48GB of RAM.
5.5 Experimental Results
Our proposed GRAM-ODE out perporrosms other baselines on all datasets in Table except for the MAPE metric in PEMS07, where it is slightly greater than that of STFGNN. ARIMA performs considerably worse than other baselines, likely because it ignores the graph structure of spatio-temporal data. GraphWaveNet performs relatively poorly, possibly due to its limited capability in stacking bi-directional random walks panding the receptive field learned from 1D CNN temporal kernels. DCRNN uses and a GRU model spatial and temporal information, respectively, but its relatively low modeling efficacy and efficiency for long-range temporal information may contribute to its subpar performance. STGCN uses GCN for the spatial domain and ID dilated convolutional operations for the temporal domain, but may lose local information due to the dilation operation, while its absence of attention-basad operations and limited capability of convolutional operations in modeling high-dinensional spatial, temporal, and spatio-lemporal correlations may also result in relatively poor performance. STSGCN captures local correlations with lo- calized spatio-temporal subgraphs but may miss global information and thus perform poorly in long-range forecasting and with data containing missing entries or noise. STFGNN uses DTW for latent spatial net- works and performs well, but is limited in learning comprehensive spatio-temporal correlations. STGODE
layers and ex-
13

---

Published in Transactions on Machine Learning Research (04/2023)
uses Neural ODEs to capture spatio-temporal dynamics and achieves very good performance compared to other baselines, but still lacks the ability to capture complex spalio-temporal correlations, balance local and global patterns, and model dynamic interactions between its components.
5.6 Case Study
We selected two nodes from the PEMS08 road networks to conduct our case study for the qualitative demonstration of results. As Fig shows, the predicted curves by our proposed GRAM-ODE (red curve) is better (more closely) aligned with the ground truth than STGODE (grey curve). The ground truth in node 109 has more fluctuations compared to the node 17 which causes more difficulty in the forecasting task. We can also observe that our model provides faster response in case of these abrupt fluctuations. This highlights the effectiveness of local ODF-GNN block in model architecture which helps the model to better learn the local fluctuations.
17-4h node
109-th node
350
140
300
120
Thaitte Hlww
250
Trafffe Flow
100
200
100
250
350
100
200
250
350
Figure The comparison of traffic flow forecasting between our proposed GRAM-ODE and STGODE visualized for node 17 (left columu) and node 109 (right column) of the PEMS08 dataset.
5.7 Ablation Study
To investigate the effect of different components of GRAM-ODE, conduct ablation experiments on PEMS04 and PEMS08 with several different variants of our model.
(1) Base: In this model, data only passes through a TCN, an ODE block and another TCN module. The ODE block only contains the global ODE-GNN with fully connected layer after that. The output of TCN is then imported to a simple fully connected layer instead of an attention module.
(2) E: Beyond (1), this model adds the dynamic edge correlations with an edge-based ODE-GNN block
and aggregates its outputs with the global ODE-GNN block through simple weighted sum operation.
(3) +L:
Beyond (2), this model adds the local ODE-GNN block with different temporal kernels to better consider the local temporal patterns. The outputs of local ODE modules are aggregated with other ODE modules through a weighted sum operation.
(4) +share: Compared to (3), this model uses shared temporal weights between the edge and global ODE- GNN modules and uses shared spatial weights among all three ODE-GNN modules, global, local and edge module.
(5) +cons: Beyond (4), this model adds an adaptive message filtering constraint to restrict the divergence of embeddings from local and global ODE modules.
(6) +agg:
Beyond (5), this model replaces the weighted sum aggregation with a newly designed aggregation module explained in Eq. (9).
(7) tres:
Beyond (6), this model adds the intra-block residual connections between outputs and inputs of the multi ODE-GNN blocks (which is explained in Eq. (20
(8) GRAM-ODE: Beyond (7), this model replaces the last. linear layer with the attention module to better
combine the features learned from different traffic graphs and parallel channels of GRAM-ODE layers
(given by Eqs. [2 and 23
14

---

Published in Transactions on Machine Learning Research (04/2023)
18 Base
17.6
Base
278
Base
E+E
27.
=+E
share
17.2
share
share
cons
agg
MAPE (9g
3g8
res
res
rOS
GRAM-ODE
16.6
GRAM-ODE
RMSE
27
GRAM-ODE
16.8
16.4
26.8
16.6
16.4
16.2
16
15.8
15.6
26.2
26
Base E
17.3
E
+
17.1
share
-shaee
share
cons
11.1
cons
26.2
485
agg
res
11
res
20
res
GRAM-ODE
GRAM-ODE
RIMSEE
MAPE
25.8
18.5
16.3
159
10.
25
157
104
288
Figure Ablation experiment results with different configurations of GRAM-ODE on PEMS03 (lop row) and PEMS08 (bottom row) datasets.
Fig. shows the results of ablation experiments with MAF, MAPE, and RMSE metries. can be observed that the edge and local ODE-GNN modulcs are both enhancing the feature representation in the traffic forecasting task. The model variant '+E' improves the performance of the base model across all metrics and both datasets. This shows that the simple node-based global ODE-GNY is insufficient in learning informative features and adding the edge-based global ODE-GNN module considerably help the perfor- mance. Without any further techniques, the model gets worse by only using '+L' beyond '+E' However, after adding the shared weight and divergence constraint techniques, the model usually gets better across all metries. The shared weights are applied in the spatial and temporal operations of global node-based and odge-based ODE-GNN blocks to take into account the dynamic correlations of edges as well as nodes in the model formulation. The constraint is added to prevent local and global ODE-GNN embeddings deviating from one another. In this figure, we can also observe the impact of aggregation layer, residual-based update layer as well as the designed attention module. It appears that, among these three elements, adding the attention module (AM) will always result in better performance, which is consistent with our hypothesis that the attention mechanism makes the model more effective in considering all the high-dimensional correlations during feature extraction.
5.8 Robustness Study
To evaluate the robustness of our model, we add noise to the historical input of training data, which can potentially mimic uncertainties and biases that can arise during the data collection process. The added noise follows zero-mean i.i.d. Gaussian distribution with fixed variance, c.g., (/(0,>2), where 72 E (2, 4). We conduct robustness analysis across different values of E (0.1, 0.3, representing the ratio of training data impacted by noise. In other words, 0 captures the performance without any noise. Fig. 6 represents the results of robustness comparisons between GRAM-ODE and STGODE on PEMS04 dataset across all the threc metrics (MAE, MAPE, and RMSE). It can be observed that GRAM-ODE performance is more robust than STGODE with different levels of added noise = 0.1; 0.3; 0.4 which is probably due to the powerful spatio-temporal feature extraction gained by multiple ODE-GN modules. We can also notice that, when the noise levels are high (with 72 4 and = 0.4), GRAM-ODE can still beat many bascline models listed in Table 3 demonstrating the significant benefit of incorporating various ODE modules in our framework which improve robustness.
15

---

Published in Transactions on Machine Learning Research (04/2023)
22
335
MAPE|N
12.5
RMMSE
78.5
155
11
355
21
D2
D3
D4
Noise Proportion
Noise Proportion
Noise Proportion
N[O.2]
N(0.4)
N(0,2)
STGODE N(0,4)
Figure 6: Robustness comparison of GRAM-ODE and STGODE on PEMS04 dataset.
6 Conclusion
In this paper, we propose Peatio-Tempora Graph Multi-ODE Neural Networks (GRAM-ODE) for forecast- ing traffic. In this model, multiple coupled ODE-GNN blocks are used to capture complex apatio-temporal dependencies from different views and learn better representations. We also add some techniques to fur- ther improve the communication between different ODE-GNN blocks including sharing weights, advanced aggregation, and divergence constraint. Extensive experiments on six real-world datasets show the superior performance of our proposed model compared to other state-of-the-art models as well as the effectiveness of cach component. Future work may focus on invostigating model compression techniques to reduce model size without sacrificing performance, exploring distributed computing strategies for elficiency, and evaluating GRAM-ODE's applicability to other spatio-temport applications like climate modeling or social network analysis.
References
Taghreed Alghamdi, Khalid Elgazzar, Magdi Bayoumi, Taysscer Sharaf, and Sumit Shah. Forccasting traffic congestion using arima modeling. In 2019 15th International Wireless Communications & Mobile Com- doi: 10.1109/I /WWCCCC2200..220000000
puling Conference (IWCMC), pp. 1227-1232, 2019.
Donald J Berndt and James Clifford. Using dynamic time warping to find patterns in time series. In KDD Seattle, WA, USA: 1994.
workshop, volume 10, pp.
George EP Box and David A Pierce. Distribution of residual autocorrelations in auloregressive-ineggaaee moving average time series models. Journal of the American statistical Association, 95(332):1509 1970.
Ming Chen, Zhewei Wei, Zengfeng Huang, Bolin Ding, and Yaliang Li. Simple and deep graph convolutional networks. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 1725-1735. PMLR, 13-18 18 Jul 2020. URL htttps:/prroceddnng: mlr htt..press//i1///eee chen2/vv html .htl
Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourl, and David K Duvenaud. Neural ordi- nary differential equations. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, vol- ume 31. Curran Associates, Inc., 2018. URL https://prooeeedings neurips. pcarrppsscc/ppppt//////eeee pdf..
Jeongwhan Choi, Hwangyong Choi, Jechyun Hwang, and Noseong Park. Graph neural controlled differential equations for traffic forecasting. Proceedings of the AAAI Conference on Artificial Intelligence, 36(6): 6367-6374, Jun. 2022. doi:
vtt.//aaii..66..2.555533 URL https://ojs. aaai org/index..inoooooooooooeoe http:/iinex..p//AA.a... php/AAAI/
article/vie=/20587
Shengdong Du, Tianrui Li, Xun Gong, Yan Yang, and Shi Jinn Horng. Traffic flow forecasting basod on hybrid deep learning framework. In 2017 12th International Conference on Intelligent Systems and Knowledge Engineering (ISKE), pp. 1-6, 2017. doi: 10.1109/ISKE220772258811
16

---

Published in Transactions on Machine Learning Research (04/2023)
Zheng Fang, Qingqing Long, Guojie Song, and Kunqing Xie. Spatial-temporal graph ode networks for traffic flow inn In Proceedings of the 27th ACM SIGKDD Conference. on Knowledge Discovery & Data Mining, KDD '21, pp. 364 373, New York, NY, USA, 2021. Association for Computing Machinery, ISBN 9781450383325. doi: 10.1145/ 10.1145/3447448.8 34474444.44477775
URL https://doi wttp//..11////4/...//// 1144//4447548. 54444443333633330 3467430
Shengnan Guo, Youfang Lin, Ning Feng, Chao Song, and Huaiyu Wan. Attention based apatial-temporal graph convolutional nešworks for traflic flow lorecasting. Proceedings of the AAAI Conference on Arlificial
Intelligence, 33(011 ):22 929, Jul. 2019. doi: 10.1609/aase L..00//aaai.33311.3330000 wtt..iiiii113011110 URL https://ojs.aaat org/
index.php/aiii wwwp/pAAaAA../ Lndex.paggAAAAA/rrtticllle////// http///arrrcccceeee//sae/
Mansura Habiba and Barak A. Pearlmutter. Neural ordinary differential equation based recurrent neural network model. 2020. URL
09807
Zhixiang He, Chi-Yin Chow, and Jia-Dong Zhang. Stenn: spatio-temporal convolutional neural network for long-term traffic prediction. In 2019 20th IEEE International Conference on Mobile Data Monagement (MDM), pp.
TEFE, 2019.
H. V. Jagadish, Johannes Gehrke, Alexandros Labrinidis, Yannis Papakonstantinou, Jignesh M. Patel, Raghu Ramakrishnan, and Cyrus Shahabi. Big data and its technical challenges. Commuun. ACM, 57(7):86- 94, jul 2014. ISSN 0001-0782, doi: 10.1145/2611567. URL https:///oi. org/10....1.
10. 1145/2611666
Young-Seon Jeong, Young-Ji Byon, Manoel Mendonca Castro-Neto, and Said M. Easa. Supervised weighting- online learning algorithm for short-term traffic flow prediction. IEEE Transactions on Intelligent Trans- portation Systems, 14(4):1700-17) 1707000777, 2013. doi: N0.1109/TTTS22233222777
Weiwei Jiang and Jiayun Luo. Graph neural network for traflic forecasting: A survey. Expert Systems with Applications, 207:117921, 2022. ISSN 0957-4174. doi:
URL
https://wwes htt.a/accc.....ooooooc../aaeaeaa
Nicola, Jones. How machine learning could help to improve climate forocasts. Nature, 5486666)))3777
Aug 2017. ISSN 1476-4687. doi: 00.1038/548379. URL
Patrick Kidger. On neural diffcrential equations. CoRR, abs/2202.024355, 2022. URL https://arxiv.com org/
abs/2202.00 02433
Patrick Kidger, James Morrill, James Foster, and Terry Lyons. Neural controlled differential equations for irregular time series. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances Neural Information Processing Systems, volume 33, pp. 6696- 6707. Curran Associates, Inc., 2020. URL httpss//proceedngs.nnnte www.nois....pppperrrrree pcc/paper/2020/ELII/ .pdf.
Patrick Kidger, James Foster, Xuechen Li, and Terry Lyons. Efficient and accurate gradients for neural SDFs. In A. Beygelzimer, Y. Dauphin, P. Liang, and Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021a. URL hhttps:/ http://wwweeeeeeeeefeummra.eaaaa
Patrick Kidger, James Foster, Xuechen Li, Harald Oberhauser, and Terry Lyons. Neural {sde}s made easy:
{SDE}s are infinite-dimensional {gan)s, 2021b. URL
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Leurning Represenlations, 2017.
Shiyong Lan, Yitong Ma, Weikang Huang, Wenwu Wang, Hongyu Yang, and Pyang Li. DSTAGNN: Dy- namic spatial-temppral aware graph neural network for traffic flow forecasting. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 11906-11917. PMLR, 17-23 Jul 2022.
Mathias Lechner and Ramin Hasani. Mixed Mixed-memory RNNs for learning torg-term dependencies in irregularly sampled time series, 2022.
17

---

Published in Transactions on Machine Learning Research (04/2023)
Mengzhang Li and Zhanxing Zhu. Spatial-temporal fusion graph neural networks for traffic flow forecasting.
Proceedings of the AAAT Conference on Artificial Intelligence, 35(5):4189-4196 May 2021. doi: 10.1609/ aaai veaiv.3555116422. CRL https://ojs.aas aaai. oogg/indes httpa/xwppp//..laaaaaraaaa/a/aaa
Qimai Li, Zhichao Han, and Xiao-Ming Wu. Deeper insights into graph convolutional networks for semi- supervised learning. In Proceedings of the Thirty-Second AAAI Conference Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth. AAAI Symposium on Educational Advances in Artificial Intelligence, AAAT18/TAAT1S/AAMI A/AAT18 AAAT Press, 2018a. ISBN
978-1-57735-800-8.
Yaguang Li, Rose Yu, Cyrus Shahabi, and Yan Liu. Diffusion convolutional recurrent neural network: Data-driven traffic forecasting, 2017. URL http://awwivv.orr////////00//110
Yaguang Li, Rose Yu, Cyrus Shahabi, and Yan Liu. Diffusion convolutional recurrent neural network: Data-driven traffic forecasting. In International Conference on Learning Representations, 2018b. URL
https:///pperrevion http://orooorm//ddSSS
Jun Liu, Amir Shahroudy, Dong Xu, and Gang Wang. Spatio-temporal Istm with trust gates for 3d human action recognition. In European conference computer vision, pp. 816 6--3.. Springer, 2016.
Antonella Longo, Marco Zappatore, Mario Bochicchio, and Shamkant B. Navathe. Crowd-sourced data collection for urban monitoring via mobile sensors. ACM Trans. Internet Technol. 18(1), oct 2017. ISSN 1533-5399. doi:
URL htttps:////// 1t1////....11/99000033 1145/3093895
James Morrill, Patrick Kidger, Lingyi Yang, and Terry Lyons Neural controlled differential equations for online prediction tasks, 2021. URL
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alchó-Bus E. Fox, and R. Garnett (eds.), Advances in Newral Informalion Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https: .pdp
7/proceedings.
wewpeediinnnnn.uuiiiee neur http://p...ppppoottt.tttaaaaaaaa
Bin Pu, Jiansong Liu, Yan Kang, Jianguo Chen, and Philip S. Yu. Mvstt: A multiview spatial-temporal transformer network for traffic-flow forecasting, IEEE Transactions on Cybernetics, pp. 1 14, 2022. doi:
10.1109/TCYB.20222 202.2.23222222311
Yulia Rubanova, Ricky TQ Chen, and David K Duvenaud. Latent ordinary differential equations for irregularly-samplee time series. Advances in neural information processing systems, 32, 2019.
Xingjian Shi, Zhourong Chen, Hao Wang, Yan Yeung, Wai-kin Wong, and Wang-chun WOO. Convolu- tional Istm network: A machine learning approach for precipitation nowcasting. C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett. (eds.), Advances in Neural Information Processing Systems, vol- ume 28. Curran Associates, Inc., 2015. URL https://proceeddngs. hwt..pps..c/ooppp////55////111/. .ddf.
Chao Song, Youfang Lin, Shengnan Guo, and Huaiyu Wan. Spatial-temporal synchronous graph convolu- tional networks: A new framework for spatial-temporal network data forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 914-921, 2020.
Yuqiao Su, Bin Ren, and Kunhua Zhang. Graph ode recurrent neural networks for traffic flow forecasting. In 2022 IEEE 5th International Conference on Electronics and Communication Engineering (ICECE), pp. 178-182, 2022.
doi: D0.100//CCCE52225.222.00008
Mascha Van Der Voort, Mark Dougherty, and Susan Watson. Combining kohonen maps with arima time series models to forecast traffic flow. Transportation Research Port C: Emerging Technologies, 4(5):307~ 318, 1996.
18

---

Published in Transactions on Machine Learning Research (04/2023)
Vivek Veeriah, Naifan Zhuang, and Guo-Jun Qi. Differential recurrent neural networks for action recognition. In Proceedings of the IEEE international conference on compuler vision, pp. 4041-4049, 2015.
Billy M Williams and Lester Hoel. Modeling and forecasting vehicular traffic flow a seasonal arima process: Theoretical basis and empirical results. Journal of transportation engineering, 129(6):664 672, 2003.
Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang, and Chengqi Zhang. Graph wavenet for deep spatial- temporal graph modeling. In Proceedings of the 28th International Joint Conference on Artificial Intelli- gence, IJCAL19, pp. 1907-1913. AAAT Press, 2021. ISBN 9780999241141.
Shi Xingjian, Zhourong Chen, Hao Wang, Dit- Yan Yeung, Wai-Kin Wong, and Wang-chun Woo. Convolu- tional lstm network: A machine learning approach for precipitation nowcasting. In Advences in neural information processing systems, pp. 802-810, 2015.
Sijie Yan, Yuanjun Xiong, and Dahua Lin. Spatial temporal graph convolutional networks for skeleton-based action recognition. In seirtg- AAAI conference on artificial intelligence, 2018.
Bing Yu, Haoteng Yin, and Zhanxing Zhu. Spatio-temporal graph convolutional networks: A deep lcarning framework for traffic forecasting. In Proceedings of the Twenly-Seventh International Joinl Conference Artificial Intelligence, IJCAI-18, pp. Organization, 7 2018. doi: 10.24963 1.22963/ijcaii
International Joint Conferences on Artificial Intelligence
Junbo Zhang, Yu Zheng, and Dekang Qi. Deep spatio-temporal residual networks for citywide crowd flows prediction. Proceedings of the AAAI Conference Artificial Intelligence, 31(1), Feb. 2017. doi: 10.1609/ anai.v31il.1033. URL https: //ojs aaai org/ index. pbtp/ALA//arrricc///////////////
Lun Zhang, Qiuchen Liu, Wenchen Yang, Nai Wei, and Decun Dong. An improved kenearest neighbor model for short-term traffic flow prediction. Procedia Social and Behanioral Sciences, 96:653 662, 2013. ISSN 1877-0428. doi: https://dooi.org/100.. http://../...ss...2110....
URL /tww:: /twp://wwwiieecweeeeettctcc com/ Intelligent and Integrated Sustainable Multimodal Trans- portation Systems Proceedings from the 13th COTA International Conference of Transportation Profes- sionals (C1CTP2013).
Jie Zhou, Ganqu Cui, Shengding Hu, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. Graph neural networks: A review of methods and applications. AT Open, 1: 57-81, 2020. ISSN 2666-6510, doi:
200.0pppe.2221000000.... URL https:
occeecedirect..com/ccciieie oeterettteeccccoocccceeeeeeea/ar
19

---

Published in Transactions on Machine Learning Research (04/2023)
Appendix
A Prediction Variance
A.1 Initialization Randomness
Table presents the average and standard deviation results of multiple runs of our model using five different random seeds. Results demonstrate that GRAM-ODE's performance exhibits only minor variance across different seeds. Moreover, despite these variations, GRAM-ODE continues to outperform other baselines across different benchmark datascts, also evident by comparing to bascline results in Table
Table 4: GRAM-ODE performance variance for different random seeds over all datasets.
Dataset
Metric
GRAM-ODE
MAE
15.72 + 0.29
PEMS03
MAPE(%)
15.98 + 0.31
RMSE
26.40 + 0.42
MAE
19.55 + 0.12
PEMS04
MAPE(%)
12.66 + 0.29
RMSE
31.05 + 0.25
MAE
21.75 = 0.30
PEMS07
MAPE(%)
9.74 0.21
RMSE
34.42 + 0.39
MAE
16.05 + 0.20
PEMSOS
MAPE(%)
10.58 + 0.13
RMSE
25.17 + 0.14
MAE
1.67 = 0.02
PEMS-BAY
MAPE(%)
3.83 = 0.03
RMSE
3.34 0.03
MAE
3.44 = 0.08
METR-LA
MAPE(%)
9.38 = 0.11
RMSE
6.64 0.05
A.2 Training Randomness
We also conducted experiments using three distinct random seeds across various cross-validation splits on the PEMS-BAY dataset (due to time constraints, we were only able to run this experiment for one dataset). The results of these experiments can be found Table As stated the manuscript, we follow the previous works and divide the datasots into train/val/test splits with a 6:2:2 ratio. Given that the data is timo-scrios, we can create different cross-validation splits for the train and (est. data as follows: T X, TX TX, XT_ XT, X T where T and X refers to train and test sets. Despite the minor variance in performance across different cross- cross- validation splits and model initialization seeds, we can observe that GRAM-ODE still outperforms other baselines for this dataset.
B Efficiency and Scalability
Figure 7 comprises two subfigures that demonstrate the efficiency and scalability of our proposed GRAM- ODE. Fig. a presents Pareto plot illustrating the trade-off between RMSE forecasting performance and inference time for various methods, including our GRAM-ODE. Although our multi-ODE framework exhibits larger inference time, it delivers superior RMSE performance compared to other methods. This highlights that, in cases where prediction accuracy is of paramount importance, the enhanced performance of our
20

---

Published in Transactions on Machine Learning Research (04/2023)
Table 5: GRAM-ODE performance variance for different cross-validation splits across different model initial- ization sceds on PEMS-BAY dataset.
CV Split
RMSE
MAE
MAPE
Soed
T. T_ X
3.39
1.7
3.95
0
3.39
1.69
3.91
3.39
1.7
3.92
2
T,X.
3.38
1.69
3.97
0
3.39
1.68
3.96
3.38
1.68
3.94
2
TT,
3.26
1.62
3.67
3.24
1.6
3.63
1
3.27
1.62
3.68
2
X,T,
2.97
1.59
3.44
0
3.00
1.63
3.49
2.98
1.59
3.45
2
_XXT
3.41
1.82
4.05
3.4
1.8
4.01
3.41
1.82
4.03
2
T.T
3.18
1.72
3.79
0
3.21
1.75
3.83
1
3.18
1.71
3.78
2
GRAM-ODE can outweigh the increased computational complexity. Fig. 7 showcases the percentage performance gain achieved by GRAM-ODE in comparison to the best-performing baseline. The dataset size on the x-axis is calculated by multiplying the squared mumber nodes by the number of timestops. As the dataset size increases, we observe that GRAM-ODE provides relatively better performance gains, indicating that our multi-ODE framework is scalable to larger datasets and nigher-dimensional inputs. This scalability further emphasizes the advantages of GRAM-ODE, especially when handling more extensive and complex data.
30.0
STGCN
0.060
PEM507
29.5
STSGCN
0.055
PEMS03
29.0
28.5
STFGNN
Relative
0.050
PEMS04
28.0
STGODE
0.045
27.5
0.040
27.0
GRAM ADE
0.035
26.5
PEM508
101
10a
10P
Inference Time (s)
Dataset Size
(a)
(b)
Figure 7: Efficiency and Scalability of GRAM-ODE. (a) RMSE and inference time trade-off for various
methods on the PEMS03 datasct. (b) Relative RMSE gain by GRAM-ODE in comparison to the best- performing baseline across datasets of varying sizes.
C Exploring the Outputs of Various ODE Modules
We analyzed the outputs of these different ODE modules in the aftermath of a traffic jam event, when fluctuations and changes more pronounced. Our observations suggest that the distinct ODE modules
21

---

Published in Transactions on Machine Learning Research (04/2023)
contribute to varying performance and predictions, likely resulting from the different features they learn from multiple perspectives. In particular, Fig- [ shows that shortly after the traffic jam, when traffic flow have sudden increases, the outputs of the Local Module (LM) appear to align more closely with the ground truth. As traffic flow stabilizes, both the Local Module (LM) and Global Module (GM) outputs exhibit better alignment with the ground truth. However, as traffic flow gradually decreases, the Global Module (GM) outputs start to diverge from the ground truth, while the Edge Module (EM) and Local Module (LM) outputs remain more consistent with the actual data. Therefore, as traffic flow changes, the aligument of the outputs with the ground truth varies across modules, indicating that each module captures different aspects of the data.
Ground Truth
4.6
EM Output
GM Output
LM Output
4.0
3.8
3.6-
09-10
09-10
09-10
09-10
09-10
09-10
09-10
09-10
09-10
08:09
08:38
09:07
09:36
10:04
10:33
11:02
11:31
12:00
(a)
0.50
GM
0.25
LM.
Figure 8: Outputs of different ODE modules during traffic fluctuations. (a) Varying performance and predictions the Local Module (LM), Glocal Module (GM), and Edge Module (EM) in response to changes in traffic flow. (b) Normalized distance of different prediction curves with ground truth at each time step.
22


Reviews:
[{"review_id": "M2pUm6rqLI", "reviewer": "Reviewer_MPh2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:** Traffic forecasting is a very interesting and challenging research topic. This paper has described the background of the topic, with a good motivation to make the graph embedding of the traffic network more informative. The proposed method is based on ordinary differential equation (ODE), which is a commonly used component in traffic forecasting. The technical part is fairly sound and not difficult to understand. The experiments contain various methods for traffic forecasting and the ablation study has verified the effect of the components. This work also considers the noisy data and evaluates the robustness of the proposed method, which I think is a very practical point and deserves research in traffic forecasting.\n\n**Weaknesses:** Despite the above merits, I have to say this paper still has space to further improve. \n1. Some descriptions are ambiguous. For example,  the authors claim that \"Despite different semantic meanings corresponding to node-based and edge-based features, they can share some important spatial and temporal patterns.\" What kind of spatial and temporal patterns are not mentioned. It could be good to clarify it with a concrete example.\n2. The novelty is not very high. The proposed method is based on ODE, which is a very common component. Some works have proposed similar approaches for traffic forecasting, such as [1] [2] below, which are not mentioned in the literature review. In [1], it even considers the local, global temporal embedding and edge-based dynamic embedding, which is very close to this work.\n[1] MVSTT: A Multiview Spatial-Temporal Transformer Network for Traffic-Flow Forecasting\n[2] Graph ODE Recurrent Neural Networks for Traffic Flow Forecasting\n3. The experimental setting is not well described. How each baseline is implemented should have been suitably mentioned.\n4. Analysis of the power of the proposed approach. Despite the evaluation metrics used, the authors have no analyses on the training and testing efficiency of the approach, which I think is important to learn the performance in a more comprehensive way.\n5. The limitations of this work and the future work are ignored.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper provides a graph convolutional network (GCN) for traffic forecasting, called GRAM-ODE. It claims the previous works ignore the local temporal embedding and edge-based dynamic embedding. Then it proposes the multi-ODE layer including the sublayers to process global temporal, local temporal and edge-based features, respectively. An attention-based aggregation layer is also applied to merge the advanced embeddings from two traffic graphs. The experiments show: 1) GRAM-ODE surpasses selective baselines; 2) the effectiveness of the key components in GRAM-ODE; 3) the robustness of the trained GRAM-ODE.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The following points are necessary to be addressed for meeting the level of acceptance. \n\n1. I strongly recommend the authors to present the analyses on the training and testing efficiency of the proposed approach. \n2. The above highly related works should be added in the literature review or excluded by some reasons.\n3. The experimental setting should be introduced with more words. The limitations and the future work should be added.\n\n", "broader_impact_concerns": "I have no concerns on the ethical implications of the work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "HJ1yfLvK7k", "reviewer": "Reviewer_LuiQ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper focuses on the task of spatio-temporal traffic forecasting. Given the recent advances of graph ODE and their limitations on the task, the authors proposed multi-ODE network (GRAM-ODE). The proposed method seems to be able to capture complex local and global dynamics from the spatial-temporal dependencies. Following are my concerns about this work, which I hope the authors can address (some of which are minor concerns):\n\n1. I would recommend the authors to add some citations in the first sentence of the Introduction. \n\n2. The different arrows in fig. 1 is barely readable.\n\n3. For the results showed in Table 3, I would suggest the authors to also report the standard deviations for a better understanding on the significances. Ideally, averaging the results of multiple runs of cross validation and reporting the average with standard deviation would be the best.\n\n4. Given the audience of this journal, I think Eq. 25 might not be necessary.\n\n5. I wonder if there are other publicly available datasets that can be used for evaluation, which can add more diversity to the datasets.\n\n6. Given the multi-ODE design of the proposed method, I think it's necessary to also compare with baselines on perspectives of scalability and efficiency. For example, complexity analysis of the proposed method, runtime comparison of the proposed method against baselines, etc.\n\n7. In the introduction, the authors mentioned that different ODE modules proposed in this work were designed for capturing different knowledge such as local temporal patterns or semantic edges. How do we validate these claims? While there are experiments showing better overall performances and ablation studies showing each designed components contributing to the performance, I think it is still hard to conclude that one design is specifically capturing some knowledge.\n\n8. Some of the different colors in Fig. 5 are barely separable, please consider changing to more separable colors or patterns for better readability.\n\n9. The manuscript seems is missing a careful proofread and typo check. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper focuses on the task of spatio-temporal traffic forecasting. Given the recent advances of graph ODE and their limitations on the task, the authors proposed multi-ODE network (GRAM-ODE). The proposed method seems to be able to capture complex local and global dynamics from the spatial-temporal dependencies. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "please address the concerns in the previous Q.", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "A3copGoLqb", "reviewer": "Reviewer_uwEo", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n* Novel architectural contributions are reasonably well-motivated\n* Apparent (but potentially marginal) improvements on an existing traffic prediction task\n* Detailed ablation study and study of model robustness to noise\n\n**Weaknesses:**\n* Experimental protocol does not support claims/hypotheses well: no variance is reported; datasets are of very limited diversity/size\n* Insufficient clarity and quality of writing and exposition\n\n**Regarding clarity of writing:**\n\nFigures:\n* Figure 2 is unclear. It is difficult to get an overview of the model.\n* Figure 3 is similarly unclear. What do the blue rectangles represent? What does “AM” stand for? What does it mean if a rectangle is shaded in a particular way (some are colored with a gradient from dark blue to blue, whereas others are from light blue to white)?\n\nWriting:\n* “To fuse these two separate sets of operations more intelligently” -> this is a vague statement: how would one determine whether a particular fusion operation is more or less intelligent?\n* The paper goes straight into very specific details around the newly introduced weight sharing, before introducing an overview of the model: I would recommend that the paper starts with a concise overview of the model and then introduces the particular novel components.\n\nLastly, there are frequent typos and grammatical inconsistencies in the paper. A few examples below:\n* Table 1 (notations): inconsistent lower-/upper-casing; “numbers of channels”\n* “Identical matrix” -> “identity matrix”\n\n\n**Regarding experiments:**\n\nExperiments:\n* Results look largely comparable to prior work. It is unclear whether benefit stems from a better choice of hyperparameters or from modeling novelties\n* No error bars or indication of variance\n* 4 small datasets from the same provider, i.e. it is unclear how this model behaves on vastly different road networks, e.g. of larger size, in different continents, cities etc.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces Graph-based Multi-ODE (GRAM-ODE), a neural network architecture for spatio-temporal traffic forecasting. The model consists of novel components for addressing temporal patterns at different scales (e.g. short-term vs. long-term) and for including semantic edge information. The model further uses multi-head attention pooling to aggregate features (while prior work in this area primarily used simpler aggregation modules such as sum- or max-pooling). The method is validated on four traffic prediction datasets and shows benefits over prior works.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* [Critical] The experimental protocol needs to be revisited: the authors should not only analyze variance of results across the different cross-validation splits, but should also report variance for different model initialization seeds, i.e. how much variance the results have based on randomness in model training/initialization. Only results that are significantly different from baselines or model variants (in terms of ablations) can be considered for testing the initial hypotheses made in the paper.\n* [Critical] The claims and hypotheses that the paper proposes need to be revisited after improving the experimental protocol. It is quite likely that several claims made in the current version of the paper will not survive this test. For example, already in the current version of the paper, the effect of special handling for local temporal patterns (“+L” ablation) is minimal or even negative. \n* [Critical] The clarity of the model definition and figures need to be significantly improved. I suggest a major revision of the method section, in line with the potential revision of the claims/hypotheses.\n* [Highly recommended] The authors should consider a wider range of datasets and/or tasks to more convincingly test their hypotheses and to show more evidence for their claims.\n* [Highly recommended] The authors should consider reducing the length of the paper to 12 pages (excl. appendix) or less; the current version of the paper is far less concise than it could be. Improving conciseness would also significantly help with clarity issues that the paper currently has.\n\n\nOther question:\nRegarding the shared spatial weight: as this is a dense matrix (with parameters initialized from a Gaussian distribution) that is being added to the adjacency matrix, this means that all message passing operations will be dense thereafter. This seems to be infeasible for large graphs, which would generally be the case for real-world applications of traffic modeling. Could the authors comment on this?\n", "broader_impact_concerns": "Not applicable. The work is an incremental architectural improvement on an existing task and model, i.e. no Broader Impact Statement is required.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers agree that the paper is useful and the ideas are interesting. Although the idea may seem straightforward, it has never been systematically studied before, and potentially is very useful for spatio-temporal modelling of signals on the graph.
Decision: Accept (poster)