Cross-Node Federated Graph Neural Network for Spatio-Temporal Data Modeling
Authors: Chuizheng Meng
Sirisha Rambhatla
Yan Liu
Venue:  (2021)
Keywords: Federated Learning
Graph Neural Network
Spatio-Temporal Data Modeling

Abstract: Vast amount of data generated from networks of sensors, wearables, and the Internet of Things (IoT) devices underscores the need for advanced modeling techniques that leverage the spatio-temporal structure of decentralized data due to the need for edge computation and licensing (data access) issues. While federated learning (FL) has emerged as a framework for model training without requiring direct data sharing and exchange, effectively modeling the complex spatio-temporal dependencies to improve forecasting capabilities still remains an open problem. On the other hand, state-of-the-art spatio-temporal forecasting models assume unfettered access to the data, neglecting constraints on data sharing. To bridge this gap, we propose a federated spatio-temporal model -- Cross-Node Federated Graph Neural Network (CNFGNN) -- which explicitly encodes the underlying graph structure using graph neural network (GNN)-based architecture under the constraint of cross-node federated learning, which requires that data in a network of nodes is generated locally on each node and remains decentralized. CNFGNN operates by disentangling the temporal dynamics modeling on devices and spatial dynamics on the server, utilizing alternating optimization to reduce the communication cost, facilitating computations on the edge devices. Experiments on the traffic flow forecasting task show that CNFGNN achieves the best forecasting performance in both transductive and inductive learning settings with no extra computation cost on edge devices, while incurring modest communication cost.

Under review as a conference paper at ICLR 2021
CROSS-NODE FEDERATED GRAPH NEURAL NETWORK FOR SPATIO-TEMPORAL DATA MODELING
Anonymous authors Paper under double-blind review
ABSTRACT
Vast amount of data generated from networks of sensors, wearables, and the In- ternet of Things (IoT) devices underscores the need for advanced modeling tech- niques that leverage the spatio-temporal structure of decentralized data due to the need for edge computation and licensing (data access) issues. While federated learning (FL) has emerged as a framework for model training without requir- ing direct data sharing and exchange, effectively modeling the complex spatio- temporal dependencies to improve forecasting capabilities still remains an open problem. On the other hand, state-of-the-art spatio-temporal forecasting mod- els assume unfettered access to the data, neglecting constraints on data sharing. To bridge this gap; propose federated spatio-temporal model Cross-Node Federated Graph Neural Network (CNFGNN)- - which explicitly encodes the un- derlying graph structure using graph neural network (GNN)-based architecture under the constraint of cross-node federated learning, which requires that data in network of nodes is generated locally on each node and remains decentralized. CNFGNN operates by disentangling the temporal dynamics modeling on devices and spatial dynamics on the server, utilizing alternating optimization reduce the communication cost, facilitating computations on the edge devices. Experiments on the traffic flow forecasting task show that CNFGNN achieves the best forecast- ing performance in both transductive and inductive learning settings with no extra computation cost on edge devices, while incurring modest communication cost.
INTRODUCTION
Modeling the dynamics of spatio-temporal data generated from networks of edge devices or nodes (e.g. sensors, wearable devices and the Internet of Things (IoT) devices) is critical for various appli- cations including traffic flow prediction (Li et al., 2018; Yu et al., 2018), forecasting (Seo et al., 2019; Azencot et al., 2020), and user activity detection (Yan et al., 2018; Liu et al., 2020). While existing works on ppatio-temporal dynamics modeling (Battaglia et al., 2016; Kipf et al., 2018; Battaglia et al., 2018) assume that the model is trained with centralized data gathered from all devices, the volume of data generated these edge devices precludes the use of such centralized data processing, and calls for decentralized processing where computations on the edge can lead to significant gains improving the latency. In addition, in case of spatio-temporal forecasting, the edge devices need to leverage the complex inter-dependencies to improve the prediction performance. Moreover, with increasing concerns about data privacy and its access restrictions due to existing licensing agree- ments, it is critical for spatio-temporal modeling to utilize decentralized data, yet leveraging the underlying relationships for improved performance.
Although recent works in federated learning (FL) (Kairouz et al., 2019) provides a solution for training model with decentralized data on multiple devices, these works either do not consider the inherent spatio-temporal dependencies (McMahan et al., 2017; Li et al., 2020b; Karimireddy et al., 2020) or only model implicitly by imposing the graph structure in the regularization on model weights (Smith et al., 2017), the latter of which suffers from the limitation of regularization based methods due to the assumption that graphs only encode similarity of nodes (Kipf & Welling, 2017), and cannot operate in settings where only fraction of devices observed during training data
(inductive learning setting). As a result, there a need for an architecture for modeling which enables reliable computation on the edge, while maintaining the data decentralized.

---

Under review as a conference paper at ICLR 2021
To this end, leveraging recent works on federated learning (Kairouz al., 2019), introduce the cross-node federated learning requirement to ensure that data generated locally at a node remains decentralized. Specifically, our architecture Cross-Node Federated Graph Neural Network (CN- FGNN), aims to effectively model the complex spatio-temporal dependencies under the cross-node federated learning constraint. For this, CNFGNN decomposes the modeling of temporal and spa- tial dependencies using an encoder-decoder model on each device to extract the temporal features with local data, and a Graph Neural Network (GNN) based model on the server to capture spatial dependencies among devices.
As compared existing federated learning techniques that rely on regularization to incorporate spatial relationships, CNFGNN leverages an explicit graph structure using graph neural network- based (GNNs) architecture, which leads to performance gains. However, the federated learning (data sharing) constraint means that the GNN cannot be trained in a centralized manner, since each node can only access the data stored on itself. To address this, CNEGNN employs Split Learning (Singh et al., 2019) to train the spatial and temporal modules. Further, to alleviate the associated high com- munication cost incurred by Split Learning, we propose an alternating optimization-ban procedure of these modules, which incurs only half the communication overhead as compared to comparable Split Learning architecture. Here, we also use Federated Averaging (FedAvg) (McMa- han et al., 2017) train shared temporal feature extractor for all nodes, which leads to improved empirical performance.
training
Our main contributions are follows:
1. We propose Cross-Node Federated Graph Neural Network (CNFGNN), GNN-based fed- erated learning architecture that captures complex spatio-temporal relationships among multiple nodes while ensuring that the data generated locally remains decentralized at no extra computation cost at the edge devices.
2. Our modeling and training procedure enables GNN-based architectures to be used in fed- erated learning settings. We achieve this by disentangling the modeling of local temporal dynamics on edge devices and spatial dynamics on the central server, and leverage an alter- nating optimization-bass based procedure for updating the spatial and temporal modules using Split Learning and Federated Averaging to enable effective GNN-based federated learning.
3. We demonstrate that CNFGNN achieves the best prediction performance (both in trans- ductive and inductive settings) at no extra computation cost on edge devices with modest communication cost, as compared to the related techniques traffic flow prediction task.
RELATED WORK
Our method derives elements from graph neural networks, federated learning and crivacy-preservve graph learning, we now discuss related works in these areas in relation to our work.
Graph Neural Networks (GNNs). GNNs have shown their superior performance on various learning tasks with graph-structured data, including graph embedding (Hamilton al., 2017), node classification (Kipf & Welling, 2017), apatio-tempora data modeling (Yan et al., 2018; Li et al., 2018: Yu et al., 2018) and multi-agent trajectory prediction (Battaglia et al., 2016; Kipf al., 2018; Li et al., 2020a). Recent GNN models (Hamilton et al., 2017: Ying et al., 2018; You et al., 2019; Huang et al., 2018) also have sampling strategies and are able scale on large graphs. While GNNs enjoy the benefit from strong inductive bias (Battaglia al., 2018; Xu et al., 2019), most works require centralized data during the training and the inference processes.
Federated Learning (FL.. Federated learning is a machine learning setting where multiple clients train a model in collaboration with decentralized training data (Kairouz et al., 2019). It requires that the raw data of each client is stored locally without any exchange or transfer. However, the decen- tralized training data comes at the cost of less utilization due to the heterogeneous distributions of data on clients and the lack of information exchange among clients. Various optimization algorithms have been developed for federated learning on non-III and unbalanced data (McMahan al., 2017; Li et al., 2020b; Karimireddy et al., 2020). Smith et al. (2017) propose a multi-task learning frame- work that captures relationships amongst data. While the above works mitigate the caveat of missing
2

---

Under review as a conference paper at ICLR 2021
neighbors' information to some extent, they are not as effective GNN models and still suffer from the absence of feature exchange and aggregation.
Alternating Optimization. Alternating optimization is popular choice in non-convex optimiza- tion (Agarwal et al., 2014; Arora et al., 2014; 2015; Jain Kar, 2017). In the context of Federated Learning, Liang al. (2020) uses alternating optimization for learning simple global model and reduces the number of communicated parameters, and He et al. (2020) uses alternating optimization for knowledge distillation from server models edge models. In our work, we utilize alternating optimization to effectively train on-device modules and the server module jointly, which captures temporal and spatial relationships respectively.
Privacy-Preserviee Graph Learning. Suzumura et al. (2019) and Mei et al. (2019) use statis- ties of graph structures instead of node information exchange and aggregation to avoid the leakage of node information. Recent works have also incorporated graph learning models with privacy- preserving techniques such Differential Privacy (DP), Secure Multi-Party Computation (MPC) and Homomorphic Encryption (HE). Zhou et al. (2020) utilize MPC and HE when learning a GNN model for node classification with vertically split data to preserve silo-level privacy instead of node- level privacy. Sajadmanesh & Gatica-Perez (2020) preprocesses the input raw data with DP before feeding it into a GNN model. Composing privacy- sivaacy-reserving techniques for graph learning can help build federated learning systems following the mrivac-in-depth principle, wherein the privacy properties degrade as gracefully as possible if one technique fails (Kairouz al., 2019).
CROSS-NODE FEDERATED GRAPH NEURAL NETWORK
3.1 PROBLEM FORMULATION
Given a dataset with graph G = (V, E), a feature tensor E R |VX.... and a label tensor Y E R(V[ x..., we consider learning a model under the cross-node federated learning constraint: node feature x; X node label y: Y, Y..... and model output ÿ, are only visible to the node i.
One typical task that requires the cross-node federated learning constraint is the prediction of spatio- temporal data generated by network of sensors. In such a scenario, V is the set of sensors and & describes relations among sensors (e.g. Eij E if and only if the distance between U; and by is below some threshold). The feature tensor æ, RRmDD represents the i-th sensor's records in the D-dim space during the past time steps, and the label y, RaxD represents the i-th sensor's records in the future n time steps. Since records collected on different sensors owned by different users/organizations may not be allowed to be shared due to the need for edge computation or li- censing issues on data access, it is necessary to design an algorithm modeling the spatio-temporal relation without any direct exchange of node-level data.
3.2 PROPOSED METHOD
We now introduce our proposed Cross-Node Federated Graph Neural Network (CNFGNN) model. Here, we begin by disentangling the modeling of node-level temporal dynamics and server-level spa- tial dynamics as follows: (i) (Figure Ic) on each node, an encoder-decoder model extracts temporal features from data on the node and makes predictions; (ii) (Figure 1b) on the central server, a Graph Network (GN) (Battaglia al., 2018) propagates extracted node temporal features and outputs node embeddings, which incorporate the relationship information amongst nodes. (i) has access to the not shareable node data and is executed on each node locally. (ii) only involves the upload and download of smashed features and gradients instead of the raw data on nodes. This decomposition enables the exchange and aggregation of node information under the cross-node federated learning constraint.
3.2.1 MODELING OF NODE-LEVEL TEMPORAL DYNAMICS
We modify the Gated Recurrent Unit (GRU) based encoder- architecture in (Cho et al., 2014) for the modeling of node-level temporal dynamics on each node. Given an input sequence a; E R on the i-th node encoder sequentially reads the whole sequence and outputs the
3

---

Under review as a conference paper at ICLR 2021
Server
GN
GN
(2)
(3)
Auj
Now
(KK
Node
Node
FodAvg: 8 Licy 8ed
((2)
(1)
(3)
Mhcn R
(a) Overview of the training procedure.
(b) Server-side Graph Network (GN).
(2)
(3)
hG
V
Li
hG
(0)
Encoder;
hC,i
hG,c,
c,
(4)
Li,0 y/ iin-2
Ii,1
Di,m
vo,0
(1) 0 c
c,
Decoder;
h hG,c,i
Vinn
V1.0
e, ((D:,8))
Vi,n 1
Vi,1
Y4.0
(c) Encoder-decoder the i-th node.
Figure 1: Cross-Node Federated Graph Neural Network. (a) In each round of training, we alternately train models on nodes and the model on the server. More specifically, we sequentially execute: (1) Federated learning of on-node models. (2) Temporal encoding update. (3) Split Learning of GN. (4) On-node graph embedding update. (b) Detailed view of the server-side GN model for modeling spatial dependencies in data. (c) Detailed view of the encoder-decoder model on the i-th node.
hidden state he. as the summary of the input sequence according to Equation
he Encoder, (0)
where (0) a zero-valued initial hidden state vector.
To incorporate the spatial dynamics into the prediction model each node, we concatenate h with the node embedding hg.c. generated from the procedure described in 3.2.2, which contains spatial information, as the initial state vector of the decoder. The decoder generates prediction ýi in an auto-regressive way starting from the last frame of the input sequence Ti,m. with the concatenated hidden state vector.
(2) We choose the mean squared error (MSE) between the prediction and the ground truth values as the loss function, which is evaluated on each node locally.
ÿ Decoders (II,, [ha, hGG...)).
3.2.2 MODELING OF SPATIAL DYNAMICS
To capture the complex spatial dynamics, adopt Graph Networks (GNs) proposed in (Battaglia et al., 2018) to generate node embeddings containing the relational information of all nodes. The central server collects the hidden state from all nodes {be,a E V} the input the GN, Each layer of GN updates the input features as follows:
e = 6" (ek, Vrg. Vest u)
e = p E
6" (e. V..
c- v'
(3)
u 8" v',
p pー (V')
4

---

Under review as a conference paper at ICLR 2021
Algorithm
Training algorithm of CNFGNN on the server side.
Server executes:
20:
(r):0) (rg-1) Initialize GN
GN
Initialize server-side GN
weights
21:
for server round = 1, Rg do
GN: client model weights ( do).d
22:
(ha,cali V) GN
v); 1 Cra "
GN
2: for each node i E V in parallel do
23:
for each client E V in parallel do
Initialize client model 00 0.0 4: Initialize graph encoding on node
24:
Vhace Sigesli ClientBackwardi
3:
i,h.....).
25:
i.a.,..bbckkwrdd
(0) h hG.e.i = GGG.e.
0ck V Vheeele) l.).
5: end for
26:
end for
6: for global round ri 1,2, Rg do
27:
Eiev
7: I/ (1) Federated learning of on-node models.
CN
(
0 (1)
28:
GN
ON 11.. V
8: for each client i parallel do Oc. f ClientUpdate(i).
9:
29:
end for
end for
30: 0
l (T... GN
11:
0e + Liey N 0c.i.
GN
31: I/ (4) On-node graph embedding up-
parallel do Initialize client model: 0 (0) end for
for each client
date.
13:
o,t O..
32:
{eG.o, li V} +
14:
GN((haale V]; 0(x))
(2) Temporal encoding update. 16:
33:
for each client E V in parallel do
for each client i E V in parallel do
34:
Set graph encoding on client as
he, + ClientEncode(i). end for
hG.e.
18:
35: end for
(3) Split Learning of GN.
36: end for
Algorithm Training algorithm CNFGNN on client side.
ClientUpdate(():
7: Oc. ((R))
1: for client round re = 1,2.... R. do
8: return o2,5 to server. ClientEncode(i):
h (Tr)
ea Encoder; 0 0:r- -)..n))
2:
3:
Encoder,(a;; Osne) to I: return ha.i
Zi.m [h (re) hG.,,ll; ("c d))c-))
server.
4:
4 + ((Di,y).
ClientBackward(i hG.e,)):
((re) (---1) no V
ÿ. < Decoder, (2 (he,s hG,e, Odec).
5:
Vo(re-1)
& +- E(D.. y)
6: end for
3: return Vha e to server.
where V,, u are edge features, node features and global features respectively. de di dh are neural networks. pr pm p" are aggregation functions such as summation. As shown in Figure 1b, we choose 2-layer GN with residual connections for all experiments. We set e Wr..,, (W the adjacency matrix). and assign the empty vector u as the input the first GN layer. The server-side GN outputs embeddings (hG,e, i E V) for all nodes, and sends the embedding of each node correspondingly.
he
3.2.3 ALTERNATING TRAINING OF NODE-LEVEL. AND SPATIAL MODELS
One challenge brought about by the cross-node federated learning requirement and the server-side GN model the high communication cost in the training stage. Since we distribute different parts of the model different devices, Split Learning proposed by (Singh al., 2019) is potential solution
5

---

Under review as a conference paper at ICLR 2021
Table 1: Statistics of datasets PEMS-BAY and METR-LA.
Dataset
# Nodes
# Directed Edges
# Train Seq
# Val Seq
# Test Seq
TEMS-BAY
325
2369
36465
5209 3425
10419
METR-LA
207
1515
23974
6850
for training. where hidden vectors and gradients are communicated among devices. However, when simply train the model end-to-end via Split Learning, the central server needs receive hidden states from all nodes and to send node embeddings to all nodes in the forward propagation, then must receive gradients of node embeddings from all nodes and send back gradients of hidden states to all nodes in the backward propagation. Assume all hidden states and node embeddings have the same size S, the total amount of data transmitted in each training round of the GN model is 4/V|S.
To alleviate the high communication cost in the training stage, we instead alternately train models on nodes and the GN model on the server. More specifically, each round of training, we (1) fix the node embedding hG... and optimize the encoder-decoder model for R. rounds, then (2) we optimize the GN model while fixing all models on nodes. Since models on nodes are fixed, he,; stays constant during the training of the GN model, and the server only needs to fetch he, from nodes before the training of GN starts and only to communicate node embeddings and gradients. Therefore, the average amount of data transmitted in each round for Rss rounds of training of the GN model reduces to 2+2R, Algorithm 2.
R. |VIS. We provide more details of the training procedure in Algorithm 1 and
To more effectively extract temporal features from each node, we also train the encoder-decoder models on nodes with the FedAvg algorithm proposed in (McMahan et al., 2017). This enables all nodes to share the same feature extractor and thus share a joint hidden space of temporal features, which avoids the potential overfitting of models on nodes and demonstrates faster convergence and better prediction performance empirically.
4 EXPERIMENTS
We evaluate the performance of CNFGNN and all baseline methods on the traffic forecasting task, which is an important application for spatio-temporal data modeling. We reuse the following two real-world large-scale datasets in (Li et al., 2018) and follow the same preprocessing procedures: (1) PEMS-BAY: This dataset contains the traffic speed readings from 325 sensors in the Bay Area over 6 months from Jan 1st, 2017 to May s1st, 2017. (2) METR-LA: This dataset contains the traffic speed readings from 207 loop detectors installed on the highway of Los Angeles County over 4 months from Mar 1st, 2012 to Jun 30th, 2012.
For both datasets, we construct the adjacency matrix of sensors using the Gaussian kernel with threshold: Wi, d if di,j >= clse 0, where di, exp (- dist(ws.,y)" ). dist(U; bty) is the road network distance from sensor v; to sensor üj. a is the standard deviation of distances and i is the threshold. We set x 0.1 for both datasets.
We aggregate traffic speed readings in both datasets into 5-minute windows and truncate the whole sequence multiple sequences with length 24. The forecasting task is predict the traffic speed in the following 12 steps of each sequence given the first 12 steps. We show the statistics of both datasets Table 1.
4.1 SPATIO-TEMPORE DATA MODELING: TRAFFIC FLOW FORECASTING
Baselines We compare CNFGNN with the following baselines. (1) GRU (centralized): a Gated
Recurrent Unit (GRU) model trained with centralized sensor data. (2) GRU + GN (centralized): a model directly combining GRU and GN trained with centralized data, whose architecture is similar to CNFGNN but all GRU modules on nodes always share the same weights. We see its performance as the upper bound of the performance of CNFGNN. (3) GRU (local): for each node we train GRU model with only the local data on it. (4) GRU + FedAvg: a GRU model trained with the Federated Averaging algorithm (McMahan al., 2017). (5) GRU + FMTL: for each node we train

---

Under review as a conference paper at ICLR 2021
GRU model using the federated multi-task learning (FMTL) with cluster regularization (Smith et al., 2017) given by the adjacency matrix. For each baseline, we have variants of the GRU model to show the effect of on-device model complexity: with 63K parameters and the other with 727K parameters. For CNEGNN, the encoder-decoder model on each node has 64K parameters and the GN model has 1M parameters.
Table 3: Comparison of the computation cost on edge devices and the communication cost. We use the amount of floating point operations (FLOPS) to measure the computational cost of models edge devices. We also show the total size of data/parameters transmitted in the training stage (Train Comm Cost) until the model reaches its lowest validation error.
Comp Cost On Device (GFLOPS)
PEMS-BAY
METR-LA
Method
Train Comm
Train Comm
RMSE
RMSE
Cost (GB)
Cost (GB)
GRU (63K) + FMTL. GRU (727K) FMTL
0.159
3.961
57.823
11,548
99.201
1.821
3,955
359.292
11,570
722.137
CNFGNN (64K + IM)
0.162
3.822
237.654
11.487
222.246
Discussion Table 2 shows the com- parison of forecasting performance and Table shows the comparison of computation cost on device and com- munication cost of CNFGNN and baselines. We make the following ob- servations. Firstly, when we compare the best forecasting performance of each baseline over the 2 GRU vari- ants, GRU trained with FedAvg per- forms the worst in terms of forecast- ing performance compared to GRU trained with centralized data and GRU trained with local data (4.432 vs 4.010/4.12 124 on PEMS-BAY and 12.058 vs 11.730/1 1.801 on METR- LA), showing that the data distribu- tions on different nodes are highly heterogeneous, and training sin- gle model ignoring the heterogeneity suboptimal.
Table 2: Comparison of performance the traffic flow forecasting task. We use the Rooted Mean Squared Error (RMSE) evaluate the forecasting performance.
Method
PEMS-BAY
METR-LA
GRU (centralized, 63K)
4.124
11.730
GRU (centralized, 727K)
4.128
11.787
GRU + GN
3.816
11.471
(centralized, 64K + 1M)
GRU (local, 63K)
4.010
11.801
GRU (local, 727K)
4.152
12.224
GRU (63K) FedAvg
4.512
12.132
GRU (727K) + FedAvg
4.432
12.058
GRU (63K) FMTL
3.961
11.548
GRU (727K) + FMTL
3.955
11.570
CNFGNN (64K + IM)
3.822
11.487
Secondly, both the GRU+FMTL baseline and CNEGNN consider the spatial relations among nodes and show better forecasting performance than baselines without relation information. This shows that the modeling of spatial dependencies is critical for the forecasting task.
Lastly, CNFGNN achieves the lowest forecasting error both datasets. The baselines that increases the complexity of on-device models (GRU (727K) + FMTL) gains slight or even no improvement the cost of higher computation cost on edge devices and larger communication cost. However, due to its effective modeling of spatial dependencies in data, CNFGNN not only has the largest improvement of forecasting performance, but also keeps the computation cost on devices almost unchanged and maintains modest communication cost compared to baselines increasing the model complexity on devices.
4.2 INDUCTIVE LEARNING ON UNSEEN NODES
Set-up Another advantage of CNEGNN is that it can conduct inductive learning and generalize to larger graphs with nodes unobserved during the training stage. We evaluate the performance of CNFGNN under the following inductive learning setting: for each dataset, we first sort all sensors based on longitudes, then use the subgraph the first 1/% of sensors train the model and evaluate
7

---

Under review as a conference paper at ICLR 2021
Table 4: Inductive learning performance measured with rooted mean squared error (RMSE).
PEMS-BAY
METR-LA
Method
25%
50%
75%
25%
50%
75%
GRU (63K) + FedAvg
4.863
4.847
4.859
11.993
12.104
12.014
CNFGNN (64K 1M)
4.541
4.598
4.197
12.013
11.815
11.676
Centralized
Centralized
0.4
0,400
SL. FodAvg
FedAvo
Vall Loss
AT. wio Fed/vgg
Vil Loss
0,375
AT, w/o FedAvg
FedAvg
FedAvg
0.3
0.350
0.325
0.2
0,300
10
20
30
40
50
60
20
40
60
80
Epoch
Epoch
(a) PEMS-BAY
(b) METR-LA
Figure 2: Validation loss during the training stage of different training strategies.
the trained model on the entire graph. For each dataset we select 17% = 25%, 50%, 75%. Over all baselines following the cross-node federated learning constraint, GRU (local) and GRU + FMTL requires training new models on unseen nodes and only GRU + FedAvg is applicable to the inductive learning setting.
Discussion Table 4 shows the performance of inductive learning of CNFGNN and GRU FedAvg baseline on both datasets. We observe that under most settings, CNFGNN outperforms the GRU + FedAvg baseline (except on the METR-LA dataset with 25% nodes observed in training, where both models perform similarly), showing that CNFGNN has the stronger ability of generalization.
4.3 ABLATION STUDY: EFFECT OF ALTERNATING TRAINING AND FEDAVG NODE-LEVEL AND SPATIAL MODELS
Baselines We compare the effect of different training strategies of CN- FGNN: (1) Centralized: CNFGNN trained with centralized data where all nodes share one single encoder-decoder.
Table 5: Comparison of test error (RMSE) and the communication cost during training of different training strategies of CNFGNN.
PEMS-BAY
METR-LA
Method
Train Comm
Train Comm
RMSE
RMSE
(2)
Cost (GB)
Cost (GB)
Split Learning (SL):
Centralized
3.816
11.471
CNFGNN trained with
split learning (Singh et al., 2019), where models on nodes and the model on the server are jointly trained by exchanging hidden vectors and gradients. (3) Split
SL SL + FedAvg
3.914
350.366
12.186
307.627
4.383
80.200
11.631
343.031
AT. w/o FedAvg
4.003
5221.576
11.912
2434.985
AT FedAvg
3.822
237.654
11.487
222.246
Learning + FedAvg (SL FedAvg): A variant of SL that synchronizes the weights of encoder-
decoder modules periodically with FedAvg. (4) Alternating training without Federated Averaging of models nodes (AT, w/o FedAvg). (5) Alternating training with Federated Averaging nodes described in Section 3.2.3 (AT FedAvg).
Discussion Figure shows the validation loss during training of different training strategies on PEMS-BAY and METR-LA datasets, and Table shows their prediction performance and the com-
8

---

Under review as a conference paper at ICLR 2021
munication cost in training. We notice that (1) SL suffers from suboptimal prediction performance and high communication costs on both datasets; SL + FedAvg does not have consistent results on both datasets and its performance is always inferior to AT + FedAvg. AT FedAvg consistently outperforms other baselines on both datasets, including variant without FedAvg. (2) AT + Fe- dAvg has the lowest communication cost METR-LA and the 2nd lowest communication cost PEMS-BAY, on which the baseline with the lowest communication cost (SL + FedAvg) has much higher prediction error (4.383 vs 3.822). Both illustrate that our proposed training strategy, SL FedAvg, achieves the best prediction performance well as low communication cost compared to other baseline strategies.
4.4 ABLATION STUDY: EFFECT OF CLIENT ROUNDS AND SERVER ROUNDS
Set-up We further investigate the effect of different compositions of the number of client rounds (R,) Algorithm and the number of server rounds (R,) Algorithm 1. To this end, we vary both Rg and Rg over [1,10,20].
Discussion Figure 3 shows the forecasting performance (measured with RMSE) and the total commu- nication cost in the training of CN- FGNN under all compositions of (Re. Rg) on the METR-LA dataset. We observe that: (1) Models with lower Re/Re, ratios (Ra/ R. < 0.5) tend have lower forecasting errors while models with higher Re/ R, ratios (Re/R, > 2) have lower communi- cation cost in training. This is be- cause the lower ratio of Re/Rg en- courages frequent exchange of node information at the expense of higher communication cost, while the higher ratio of Re/ Rg acts in the op- posite way. (2) Models with similar Re/ R, ratios have similar communi- cation costs, while those with lower Rs values perform better, corroborating our observation in (1) that frequent node information ex- change improves the forecasting performance.
(1, 20)
0.5 Ro/Rs 2
600
R./R, < 0.5
Comm Cost (GB)
R./R,
(1, 10)
400
(10, 10)
200
(10, 20)
[20, 20)
(20,
(20, 1) (10, 1)
0
11.50
11.55
11.60 Forecasting RMSE
11.65
11.70
11.75
Figure 3: Effect of client rounds and server rounds (Re, R. on forecasting performance and communication cost.
CONCLUSION
We propose Cross-Node Federated Graph Neural Network (CNFGNN), which bridges the gap be- tween modeling complex spatio-temporal data and decentralized data processing by enabling the use of graph neural networks (GNNs) in the federated learning setting. We accomplish this by de- coupling the learning of local temporal models and the server-side spatial model using alternating optimization of spatial and temporal modules based on split learning and federated averaging. Our experimental results traffic flow prediction on two real-world datasets show superior performance as compared to competing techniques. Our future work includes applying existing GNN models with sampling strategies and integrating them into CNFGNN for large-scale graphs, extending CN- FGNN to fully decentralized framework, and incorporating existing privacy-preservin methods for graph learning CNFGNN, to enhance federated learning of spatio-temporal dynamics.
REFERENCES
Alekh Agarwal, Animashree Anandkumar, Prateek Jain, Pranceth Netrapalli, and Rashish Tandon. Learning sparsely used overcomplete dictionaries. Conference on Learning Theory, pp. 123- 137, 2014.

---

Under review as a conference paper at ICLR 2021
Sanjeev Arora, Rong Ge, and Ankur Moitra. New algorithms for learning incoherent and overcom- plete dictionaries. In Conference on Learning Theory. pp. 779-806, 2014.
Sanjeev Arora, Rong Ge, Tengyu Ma, and Ankur Moitra. Simple, efficient, and neural algorithms for sparse coding. 2015.
Omri Azencot, N Benjamin Erichson, Vanessa Lin, and Michael W Mahoney, Forecasting sequential data using consistent koopman autoencoders. In ICML., 2020.
Peter Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, et al. Interaction networks for learning about objects, relations and physics. In Advances in neural information processing systems. pp. 4502 4510, 2016.
Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanzhez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv:1806.000 01261, 2018.
Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. Learning phrase representations using mn encoder-decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1724-17344 2014.
Will Hamilton, Zhitao Ying. and Jure Leskovec. Inductive representation learning on large graphs. In Advances in neural information processing systems, pp. 1024-1034, 2017.
Chaoyang He, Salman Avestimehr, and Murali Annavaram. Group knowledge transfer: Collabora- tive training of large on the edge. arXiv preprini arXiv:2007.14513, 2020.
Wenbing Huang, Tong Zhang, Yu Rong, and Junzhou Huang. Adaptive sampling towards fast graph representation learning. In Advances in neural information processing systems, pp. 4558-4566 2018.
Prateek Jain and Purushottam Kar. Non-convex optimization for machine learning. Foundations and Trends()) in Machine Learning, 10(3-4):142-363. 2017. ISSN 1935-8237. doi: 10.1561/ 2200000058. URL http: /ttp:://dd..e doi wwwooo../.......... 1561 22000000500
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Keith Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. arXiv preprint
2019.
Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian U Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In Proceedings of the 37th International Conference on Machine Learning, 2020.
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. In International Conference on Learning Representations (ICLR), 2017.
Thomas N Kipf, Ethan Fetaya, Kuan-Chiel Wang. Max Welling, and Richard S Zemel. Neural relational inference for interacting systems. In /CML,, 2018.
Max Guangyu Li, Bo Jiang, Hao Zhu, Zhengping Che, and Yan Liu. Generative attention networks for multi-agent behavioral modeling. AAAI, 2020a.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. In Proceedings of the 3rd MLSys Conference, 2020b.
Yaguang Li, Rose Yu, Cyrus Shahabi, and Yan Liu. Diffusion convolutional recurrent neural net- work: Data-driven traffic forecasting. In International Conference on Learning Representations (ICLR 18), 2018.
Paul Pu Liang, Terrance Liu, Liu Ziyin, Ruslan Salakhutdinov, and Louis- Morency. Think locally, act globally: Federated learning with local and global representations. arXiv preprint arXiv:2001.01523, 2020.
10

---

Under review as a conference paper at ICLR 2021
Ziyu Liu, Hongwen Zhang, Zhenghao Chen, Zhiyong Wang. and Wanli Ouyang. Disentangling and unifying graph convolutions for skeleton-base based action recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 143-152, 2020.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-fffcieni learning of deep networks from decentralized data. In Artificial Intelli- gence and Statistics, pp. 1273-1288 1282. PMLR, 2017.
Guangxu Mei, Ziyu Guo, Shijun Liu, and Li Pan. Sgnn: A graph neural network based federated learning approach by hiding structure. In 2019 IEEE International Conference Big Data (Big Data), pp. 2560-2568. IEEE, 2019.
Sina Sajadmanesh and Daniel Getica-Perez. When differential privacy meets graph neural networks. arXiv preprint w:Xi:22006005335 2020.
Sungyong Seo, Chuizheng Meng. and Yan Liu. Physics-aware difference graph networks for sparsely-observed dynamics. In International Conference on Learning Representations, 2019.
Abhishek Singh, Praneeth Vepakomma, Otkrist Gupta, and Ramesh Raskar. Detailed compar- ison of communication efficiency of split learning and federated learning. arXiv preprint 190.0919, 2019.
Virginia Smith, Chao-Kat Chiang, Maziar Sanjabi, and Ameet S Talwalkar. Federated multi-task learning. Advances in Neural Information Processing Systems, pp. 4424-4434, 2017.
Toyotaro Suzumura, Yi Zhou, Natahalie Barcardo, Guangnan Ye, Keith Houck, Ryo Kawahara, Ali Anwar, Lucia Larise Stavarache, Daniel Klyashtomy, Heiko Ludwig, et al. Towards federated graph learning for collaborative financial crimes detection. arXiv preprint arXiv:1909 12946, 2019.
Keyulu Xu, Jingling Li, Mozhi Zhang, Simon S Du, Ken-ichi Kawarabayashi, and Stefanie Jegelka. What neural networks reason about? In International Conference Learning Representa- tions (ICLR), 2019.
Sijie Yan, Yuanjun Xiong, and Dahua Lin. Spatial temporal graph convolutional networks for skeleton-based action recognition. In AAAI, 2018.
Rex Ying, Ruining He, Kaifeng Chen, Pong Eksombatchai, William L Hamilton, and Jure Leskovec.
Graph convolutional neural networks for web-scale recommender systems. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 974 983, 2018
Jiaxuan You, Rex Ying, and Jure Leskovec. Position-aware graph neural networks. In Proceedings of the 36th International Conference on Machine Learning, 2019.
Bing Yu, Haoteng Yin, and Zhanxing Zhu. Spatio-temporal graph convolutional networks: A deep learning framework for traffic forecasting, In Proceedings of the 27th International Joint Confer- ence on Artificial Intelligence (IJCAI), 2018.
Jun Zhou, Chaochao Chen, Longfei Zheng, Xiaolin Zheng, Bingzhe Wu, Zigi Liu, and Li Wang. Privacy-preseriing graph neural network for node classification. arXiv preprint 1105 2020.

---

Under review as a conference paper at ICLR 2021
A APPENDIX
A.1 DETAILED EXPERIMENT SETTINGS
Unless noted otherwise, all models are optimized using the Adam optimizer with the learning rate le-3.
GRU (centralized) Gated Recurrent Unit (GRU) model trained with centralized sensor data. The GRU model with 63K parameters is I-layer GRU with hidden dimension 100, and the GRU model with 727K parameters is 2-layer GRU with hidden dimension 200.
GRU (local) We train one GRU model for each node with the local data only.
GRU + FedAvg We train single GRU model with Federated Averaging (McMahan et al., 2017). We select as the number of local epochs.
GRU FMTL We train GRU model for each node using the federated multi-task learning (FMTL) with cluster regularization (Smith et al., 2017) given by the adjacency matrix. More specif- ically, the cluster regularization (without the 1.2-norm regularization term) takes the following form:
R(W, .2) Xtr (tr?
(A1)
Given the constructed adjacency matrix A, J [V] (D- | L, where D is the degree matrix and L is the Laplacian matrix. Equation Al can reformulated as:
R(W, Xtr(WS(W7) X
tr(WLWT)
tr
M
A
Σ W? - -E w,ayy
lr
w
(A2)
IEV
J
i7i
0i (w;, Wi w,)).
NEV
We implement the cluster regularization via sharing model weights between each pair nodes connected by edge and select A = 0.1.
We use a GRU-based encoder-decoder model as the model on nodes, which has 1 GRU layer and hidden dimension 64. We use 2-layer Graph Network (GN) with residual connections as the Graph Neural Network model on the server side. We use the same network architecture for the cdge/node/glooal update function in each GN layer: multi-layer perceptron (MLP) with 3 hidden layers, whose sizes are [256, 256, 128] respectively. We choose R. = 1, R, = 20 for experiments on PEMS-BAY and R. 1, R. = 1 for METR-LA.
CNFGNN
A.2 CALCULATION OF COMMUNICATION CoST
We denote R the number of communication rounds for model to reach the lowest validation emor in the training stage.
GRU FMTL Using Equation A2, in each communication round, each pair of nodes exchange their model weights, thus the total communicated data amount is calculated as:
R x #nonself directed edges x size node model weights.
(A3)
CNFGNN (AT FedAvg) In each communication round, the central server fetches and sends back model weights to each node for Federated Averaging, and transmits hidden vectors and gradients for Split Learning. The total communicated data amount calculated as:
R (#nodes size of node model weights 2
(A4)
+ (1 + 2 * server round + 1) x #nodes x hidden state size).
12

---

Under review as a conference paper at ICLR 2021
CNFGNN (SL) In each communication round, each node sends and fetches hidden vectors and graidents twice (one for encoder, the other for decoder) and the total communicated data amount is:
R x 2 x #nodes hidden state size.
(A5)
CNFGNN (SL FedAvg) Compared to CNFGNN (SL), the method has extra communcation cost for FedAvg in each round, thus the total communicated data amount is:
R (#nodes size of node model weights x 2 #nodes hidden state size).
(A6)
CNFGNN (AT, w/o FedAvg) Compared to CNFGNN (AT FedAvg), there is no communcation cost for the FedAvg part, thus the total communcated data amount is:
R (1+2+ server round + 1) #nodes hidden state size.
(A7)
Table Al: Parameters used for calculating the communication cost GRU + FMTL.
Method
GRU (63K)
FMTL
GRU (727K) + FMTL
Node Model Weights Size (GB)
2.347E-4
2.708E-3
#Nonself Directed Edges
2369
PEMS-BAY
104
56 359.292
Train Comm Cost (GB)
57.823
#Nonself Directed Edges
1515
METR-LA
279 99.201
176 722.137
Train Comm Cost (GB)
Table A2: Parameters used for calculating the communication cost of CNFGNN (AT + FedAvg).
Node Model
2.384E-4
Weights Size (GB)
#Nodes
325 2.
Hidden State Size (GB)
2.173E-3
PEMS-BAY
Server Round
20
Train Comm Cost (GB)
237.654
#Nodes
207
Hidden State Size (GB)
1.429E-3
METR-LA
Server Round
R
46
Train Comm Cost (GB)
222.246
A.3 INDUCTIVE LEARNING
We have added results using 90% and 5% data on both datasets and we show the table of inductive learning results as Table A6. We observe that: (1) With the portion of visible nodes in the training stage increasing, the prediction error of CNFGNN decreases drastically. However, the increase of the portion of visible nodes has negligible contribution to the performance of GRU + FedAvg after the portion surpasses 25%. Since increasing the ratio of seen nodes in training introduces more complex relationships among nodes to the training data, the difference of performance illustrates that CNEGNN has stronger capability of capturing complex spatial relationships. (2) When the ratio of
13

---

Under review as a conference paper at ICLR 2021
Table A3: Parameters used for calculating the communication cost of CNFGNN (SL).
#Nodes
325
Hidden State Size (GB)
2.
2.173E-3
PEMS-BAY
R
31
Train Comm Cost (GB)
350.366
#Nodes
207
Hidden State Size (GB)
1.429E-3
METR-LA
65
Train Comm Cost (GB)
307.627
Table A4: Parameters used for calculating the communication cost of CNEGNN (SL + FedAvg).
Node Model
2.384E-4
Weights Size (GB)
#Nodes
325
Hidden State Size (GB)
2.173E-3
PEMS-BAY
Train Comm Cost (GB)
80.200
#Nodes
207
Hidden State Size (GB)
1.429E-3
METR-LA
71
Train Comm Cost (GB)
343.031
Table A5: Parameters used for calculating the communication cost of CNEGNN (AT, w/o FedAvg).
#Nodes
325
Hidden State Size (GB)
2.173E-3
PEMS-BAY
Server Round
20
R
44
Train Comm Cost (GB)
5221.576
#Nodes
207
Hidden State Size (GB)
1.429E-3
METR-LA
Server Round
R
49
Train Comm Cost (GB)
2434.985
5%
25%
50%
75%
90%
(a) PEMS-
5%
25%
50%
75%
90%
(b) METR-LA
Figure A1: Visualization of subgraphs visible training under different ratios.
14

---

Under review as a conference paper at ICLR 2021
Table A6: Inductive learning performance measured with rooted mean squared error (RMSE).
Mathod
PEMS-BAY
METR-LA
CNEGNN (BAK IM)
3.869
4.341
4.598
4.197
3.42
13.931
12/013
11.815
11,676
11,099
visible nodes in training extremely low (5%), there is not enough spatial relationship information in the training data to train the GN module in CNFGNN, and the performance of CNFGNN may not be ideal. We visualize the subgraphs visible training under different ratios Figure Al. However, as long the training data covers moderate portion of the spatial information of the whole graph, CNFGNN can still leverage the learned spatial connections among nodes effectively and outperforms GvU+FedAAv. We empirically show that the necessary ratio vary for different datasets (25% for PEMS- BBA and 50% for METR-LA).
A.4 THE HISTOGRAMS OF DATA ON DIFFERENT NODES
We show the histograms of traffic speed on different nodes of PEMS-BAY and METR-LA in Fig- ure A2. For each dataset, we only show the first 100 nodes ranked by their IDs for simplicity. The histograms show that the data distribution varies with nodes, thus data on different nodes are not independent and identically distributed.
15

---

Under review as a conference paper at ICLR 2021
ALT
ESSE SAS
BASE BEEE
(a) PEMS-BAY
ETT TTI
1111
(b) METR-LA
Figure A2: The histograms of data the first 100 nodes ranked by ID.
16


Reviews:
[{"review_id": "XqISMPcgCP5", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Graph neural networks and federated learning are both promising directions of works individually. This papers is apparently one of the first few attempts to combine them for spatio-temporal data modeling. The time series data in the local nodes is modelled by an Encoder-Decoder architecture and spatial locality property of various nodes is captured by the server. The Encoder at each node projects the time series data into an embedding space. This embedding is used by the GNN at the server as node features. The server side GNN outputs node embeddings. The Encoder embeddings and the GNN embeddings are then concatenated and fed to the decoder that predicts the outputs for the subsequent time steps. To ensure that all the nodes encode their temporal data in a common space, the encoders are shared by the clients. Overall, the results look promising.\n\nFollowing are some doubts / concerns I have:\n\n1. In Eq. 3, the specifics of how the edge features and global features are extracted is never mentioned in the paper. All that is mentioned is how they connect two nodes based on their distance of separation.\n\n2. Can authors please elaborate more on Step 25 of Algorithm 1? Given \\delta h_{G,c,i}, and l_i, can we compute \\delta \\theta_{G_N} Overall, how the parameters of GN are updated is not very clear to me.\n\n3. In Algorithm 2, why is the client sending gradient of node embeddings back to the server?\n\n4. In Table 3, why are we not comparing the computational cost on the server?", "extra_scores": {}, "extra_text": {}}, {"review_id": "Saqmoa4vEoq", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents an algorithm to model the complex spatio-temporal dependencies across multiple nodes under a federated learning setting. Specifically, the authors employ a GRU-based encoder-decoder architecture to construct the node-level temporal dynamics on each node and utilize a GCN to capture server-level spatial dynamics on the server. \n\nThe model consists of four major steps. First, each node encodes the local data and sends the extracted temporal vectors to the server. The server then collects all temporal vectors, builds the graph, extracts the spatial vectors, and broadcasts the server-level dynamics. Next, each node combines the node-level temporal dynamics and the server-level spatial dynamics to decode the prediction.  Finally, the backpropagation algorithm is performed. \n\nPros -------------\n\n- It is interesting to extend the concept of federated learning to graph-based spatio-temporal forecasting tasks. \n- The paper is well-written and structured.\n- Split learning seems to be a good idea to reduce communication costs.\n\nCons  -------------\n\n- Privacy concern: This paper introduces some privacy-preserving methods in Related Work, but the corresponding protection methods do not present in the paper. Or the author regards the extracted feature vectors and corresponding gradients are not sensitive. \n\n- Communication costs: Although the model makes several adjustments, the amount of data transmitted in a complete training phase is still very large (200+GB), which places a demanding requirement on the capability of edge devices. In addition, I am curious about where communication costs mainly come from. the vectors and gradients from SplitNN? or the model parameters from FedAvg?\n\n- Experiments:  An important baseline is missing, a model that aggregates all the data and directly combines GRU with GNN to predict the traffic flow without a federated setting (like DCRNN). In my view, this model should be the upper bound of the model performance, just like what the author did in section 4.3-(1). Besides, there is no explanation for the little change in the performance of GRU+FedAvg in section 4.2, what if we use more data like 90% or fewer data like 5%. Besides, the main difference between GRU+FedAvg and CNFGNN is whether to use GNN to capture spatial dependencies.  Is it spatial dependencies as the key reason that achieves better performances in an inductive learning setting?", "extra_scores": {}, "extra_text": {}}, {"review_id": "L3G-0Pjkxk_", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The proposed framework is demonstrated with two datasets from the transportation sensor\nnetwork. The experimental results on model performance and training property demonstrate the\nefficacy of the proposed method empirically. The model used is from existing works, but the\ntraining framework is relatively novel in the FL community.\nStrengths\n1. It is very encouraging to support advanced models for federated learning. The author\ngoes in the right direction. As far as I know, this is the first time that GCN+GRU is\ndiscussed under the context of federated learning.\n2. The techniques discussed in this work is comprehensive since the authors introduce\nboth the model and the training method.\nWeakness\n***1. Motivation is not well-discussed***\n(1) From the introduction section, the author claim that the node-level datasets cannot be\ncentralized. But in practice, at least sensors in IoT networks CAN be centralized to edge\nservers.\n(2) Especially, the datasets used in experiments does not have any privacy issue. It is very\nstrange to me why we should concern the privacy of the *traffic speed* from road network\nsensors. We can check the road network speed in real-time using Google Map. In other words,\nthe speed of a specific sensor is not sensitive and can be acquired by public service. Currently,\nthe Google Map example demonstrates that the road network sensors can be uploaded to\nservers from transportation agencies without any privacy concerns.\n(3) Another serious problem is that the DNN-based model (encoder-decoder used in the\nproposed method) are not deployable in the sensor device due to computational resource\nconstraint. In practice, the number of road network sensor in transportation is huge, so it is\nimpractical to ask for an expensive upgrading which requires to install a powerful DNN enabled\nchips.\n***2. No technical contribution***\nThe focus of this work is diverse (both model and training), but lack of contribution in each\naspect.\n(1) The authors argue the contribution of modeling (e.g., 4.1 modeling spatial dependencies, 4.2\n“inductive v.s. transductive”). However, for the modeling part, I believe the contribution is from\nGCN+GRU-based modeling, which is already there in many data mining publications (Equation\n(1)(2)(3) are not newly proposed models). Please refer to [1][2]. The model used in this paper is\na simplified version of these two GCN+RNN-based models in transportation. More advanced\nmodels following these two can obtain better performance than the model used in this paper.\n[1] Spatio-Temporal Graph Convolutional Networks: A Deep Learning Framework for Traffic\nForecasting\n[2] Diffusion convolutional recurrent neural network: Data-driven traffic forecasting\n(2) The authors also claim the benefit of alternating optimization (4.3, 4.4). It is good to know\nsuch an alternating method works well in practice. However, the author does not provide any\nin-depth analysis or comprehensive experimental evidence (see my comments 3 below for\ndetails).\nIn short, I suggest the author focuses on one aspect of federated learning to consolidate the\ncontribution. At least, the current version raises my concern that this work is a simple\ncombination of many techniques without meta-contribution in each one. If the overall\ncharacteristics are good, it should be encouraged. However, the effectiveness is not obvious:\nhigher accuracy, security, fairness, privacy, communication, and computation efficiency, I\ncannot see any of these advantages of such a combination:\n[Model Performance] In terms of higher accuracy, as mentioned in (2), the model used is not\nnovel. More advanced models can obtain higher accuracy. See following works of 2-(2) [1][2] for\ndetails.\n[Optimization] The experimental results of the alternating optimization has some\ncounter-intuition results. See 3 for details.\n[Security/Privacy] As for security and privacy, the hidden vector exchange may leak privacy but\nthis work does not discuss it. See 8 for details.\n[Computational Efficiency] The model used in edge devices are too computational expensive for\nresource constrained sensors. See 4 for details.\n***3. The training Algorithm is novel but the experimental result is not convincing***\nThe training method is relatively novel in federated learning. However, the authors should either\nprovide enough intuition to explain why this work (any related works in optimization theory?) or\ndemonstrate it with experimental design. The author attempts to prove that alternating training\nworks better but it’s not convincing. In Section 4.3, all the experimental results are not\nconvincing or counterintuitive:\nFirst, I am surprised that the authors have a result to say that split learning performs worse than\ncentralized training. In the essence of optimization, split learning is the same as centralized\ntraining. The only difference is that split learning offloads part of the model architecture to the\nserver-side to reduce the communication/computation cost.\nSecond, due to the first reason, that “AT w/o FedAvg” and “AT + FedAvg” works better than SL\ncan NOT demonstrate the effectiveness of alternating optimization. As I mentioned previously,\nto prove the effectiveness of alternating optimization, we need more analysis or comprehensive\nexperimental design.\nThird, another counterintuitive result is that AT + FedAvg performs better than centralized\ntraining (the red curve is lower than the blue curve). Normally, in federated learning, one should\nbelieve that centralized training is the lower bound of training/validation errors. An alternating\nlocal SGD method normally leads to variance bias in the training process, thus I believe it’s\nimpossible to obtain a better performance than centralized training.\nThe last concern is that the experiments only perform at one dataset (METR-LA). Please also\ncheck whether the same conclusion holds in another dataset.\n***4. ​Communication and computational cost***\nThe computational cost is evaluated, but what’s the implication of the number showing in terms\nof FLOPS? I don’t think the resource constrained road network sensor can handle so many\nFLOPS. Does any running time result in a real-world sensor device? If testing the sensor is\nprohibited, please provide the running time result at a low-performance smartphone or other IoT\ndevices (NVIDIA edge GPU devices).\n***5. ​Overall training time***\nThe overall training time and the bandwidth of exchanging hidden vectors should be mentioned.\nIf the network size of the exchange information is too high and the bandwidth is limited in\nwireless communication (e.g., road sensor networks), a single iteration will cost too much time\nor failure due to communication protocol constraint (e.g., in IoT setting, maximum payload\nlength is only 65535 bytes), which will lead to a long training time or impractical assumption.\nPlease discuss the size of exchange information (hidden vector/gradient) and the total training\ntime in revision.\n***6. Scalability issue is ignored***\nThe proposed method does not use a client sampling strategy, a common practice in\ncross-device FL (see the original FedAvg paper [1]), to mitigate the scalability issue. What the\nperformance if we want to learn 10 thousand sensors? Especially in the transportation setting,\nthe scalability is very important. However, the authors do not discuss this.\n[1] Communication-Efficient Learning of Deep Networks from Decentralized Data.\nhttps://arxiv.org/abs/1602.05629\n***7. Dataset: Non-I.I.D. is not properly discussed***\nA key assumption of Federated Learning is that datasets across devices are non-I.I.D. This is\nlargely ignored by the proposed methods. Slightly mentioning the heterogeneous property is not\nenough. Please discuss the details of this in revision (e.g., show the distribution diagram). More\nsignificantly, DNN models like encoder-decoder architectures normally eat a lot of samples, but\nthe number of samples in each node is small in practice. The assumption that each node has\nenough dataset to train a good model is too strong (we cannot assume the edge device has the\nstorage capability to store months of time series data since the storage ability is limited at the\nedge, ***4-6 months*** as the author mentioned). Maybe the alternating optimization method\nhelps, but it lacks analysis. In my opinion, it’s more interesting and practical to construct a\ngraph-level federated learning method since multiple nodes can be centralized to an edge\nserver belongs to a company or an agency like the transportation scenario.\n***8. ​Privacy***\nThe authors discuss some privacy-preserving methods in related works, but the proposed\nmethod does not include any privacy design. For example, [1] analyze potential leakage from\nthe hidden vectors, which proves that the proposed method does not have privacy advantages\nthan exchanging gradient/models.\n[1] Cronus: Robust and Heterogeneous Collaborative Learning with Black-Box Knowledge\nTransfer. https://arxiv.org/abs/1912.11279\n***9. Lack of Related works***\nSince alternating optimization (training algorithm) is the novel part of the proposed method,\nthere is a lack of literature review of this method, either in the general ML optimization literature,\nor the specific federated learning publications. Without such a comparison, the novelty is not\nconvincing.\n***10. Some misunderstandings of federated learning***\n(1) In the first paragraph of the introduction, the authors claim that decentralized training can\nimprove latency, but this argument is vague. Improving training latency or inference latency?\nFederated learning will cost time to train on devices, which requires a long time due to many\nrounds of communication synchronization. For the inference, to improve the latency, we need\nmodel compression techniques, which is also not the main goal of the proposed method.\n(2) The term “decentralized training” is misused. In federated learning, decentralized training\nalso refers to training using a decentralized topology (no central server). Note that FedAvg used\nin the proposed method is a centralized training with decentralized datasets. Please clarify the\ndifference in revision, making sure the readers understand which aspect is decentralized, the\ndatasets, the distributed optimization algorithm, or just computation.\n***11. Reproducibility***\nThe proposed algorithm is complex. Without the code release, it’s hard to check the correctness\nof the implementation. Deep learning normally costs time, I am curious to know how the authors\nimplement such a complex system/algorithm which can train so many nodes in parallel. If it is a\nsimulator, the training time will be too long since so there so many nodes (300+) with DNN\nmodels. If the authors can provide code, maybe all my confusion can be addressed, and I am\nwilling to increase my rating.\n***Suggestions***\n(1) I encourage the authors demonstrate the idea with more realistic datasets. For example,\nin social network, it is possible to run models on the smartphones of Internet users to\nprotect their privacy, which is much better than the transportation example.\n(2) In the current version, there is no contribution in modeling. Thus, I suggest authors to\nthink about customizing the model to obtain benefit of efficiency and privacy.\n(3) More intuition and discussion are requires for the training method. Why alternating\nminimization works well in the GCN setting.\n(4) Since the training results are not convincing, I suggest authors run more experiments to\nunderstand the proposed method and explain why there are some counter-intuitive\nresults.\n(5) Privacy is not the main focus, but some discussions and related works should be\nmentioned.\nOverall Comments\nDue to so many concerns mentioned above, I encourage the authors to have a deeper\nunderstanding of federated learning and address these issues in the revision, especially the\nmotivation, and demonstrating the effectiveness of the proposed alternating training method.\nAfter thinking a while, I cannot find a practical scenario that requires GCN-based node-level\nprivacy. At least, the sensor dataset is not a good example. Maybe the authors can search for a\npractical dataset to demonstrate motivation.\nIn addition, please focus on just one gist rather than intertwine too many aspects and claim all\ncontributions without any trade-off discussion. Besides the training framework, it’s also\ninteresting research if we can see newly developed model architectures can trade-off many\nimportant aspects in FL.", "extra_scores": {}, "extra_text": {}}, {"review_id": "0n3ZzRoj7HO", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a new modeling framework which would utilize the spatio-temporal information in the data by developing a Cross-Node-Federated-Graph-Neural-Network (CNFGNN). The methodology seems to be clearly explained while it relies on many existing methods in the literature which limits the novelty of the paper. The model is applied to two real data sets and outperforms some existing methods. Overall, I find the methodology interesting.\n\nSome comments:\n\nIn the traffic flow forecasting data: (1) Have you tried different weights (W_{i,j}) instead of the Gaussian kernel function? Does the performance measurements change by using a different weight function? It would be appropriate to show the robustness of the method with respect to changes in the spatial weight selection. (2) Removing trend and seasonality is a common practice in time series forecasting, but it is not clear whether this step is addressed properly or not. What about the seasonality in the data? Is this issue resolved in the preprocessing? ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper received mixed reviews: two positives (6, 6) and two negatives (5, 3). However, the positive reviewers have very low confidence, do not show strong supports for this paper. The reviewers raised various concerns about this paper, and there still exist remaining critical issues although the authors made substantial efforts to answer the questions.

After reading the paper and all the comments by the reviewers, I decided to recommend rejecting this paper mainly due to its weak technical contribution and ignorance of privacy issues. Note that this opinion is shared with two negative reviewers. The proposed model and alternative training scheme are straightforward, and the novelty is not distinct. Also, the authors seem to assume that "the extracted feature vectors and corresponding gradients are not sensitive". This comment is given by R2 but has not been clarified. The proposed method is lacking in this aspect and it is hard to say that it is an FL approach.


Decision: Reject