Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Graph neural networks and federated learning are both promising directions of works individually. This papers is apparently one of the first few attempts to combine them for spatio-temporal data modeling. The time series data in the local nodes is modelled by an Encoder-Decoder architecture and spatial locality property of various nodes is captured by the server. The Encoder at each node projects the time series data into an embedding space. This embedding is used by the GNN at the server as node features. The server side GNN outputs node embeddings. The Encoder embeddings and the GNN embeddings are then concatenated and fed to the decoder that predicts the outputs for the subsequent time steps. To ensure that all the nodes encode their temporal data in a common space, the encoders are shared by the clients. Overall, the results look promising.

Following are some doubts / concerns I have:

1. In Eq. 3, the specifics of how the edge features and global features are extracted is never mentioned in the paper. All that is mentioned is how they connect two nodes based on their distance of separation.

2. Can authors please elaborate more on Step 25 of Algorithm 1? Given \delta h_{G,c,i}, and l_i, can we compute \delta \theta_{G_N} Overall, how the parameters of GN are updated is not very clear to me.

3. In Algorithm 2, why is the client sending gradient of node embeddings back to the server?

4. In Table 3, why are we not comparing the computational cost on the server?
review_title: The paper is okay to me

final_score: rating: 3: Clear rejection
review_content: The proposed framework is demonstrated with two datasets from the transportation sensor
network. The experimental results on model performance and training property demonstrate the
efficacy of the proposed method empirically. The model used is from existing works, but the
training framework is relatively novel in the FL community.
Strengths
1. It is very encouraging to support advanced models for federated learning. The author
goes in the right direction. As far as I know, this is the first time that GCN+GRU is
discussed under the context of federated learning.
2. The techniques discussed in this work is comprehensive since the authors introduce
both the model and the training method.
Weakness
***1. Motivation is not well-discussed***
(1) From the introduction section, the author claim that the node-level datasets cannot be
centralized. But in practice, at least sensors in IoT networks CAN be centralized to edge
servers.
(2) Especially, the datasets used in experiments does not have any privacy issue. It is very
strange to me why we should concern the privacy of the *traffic speed* from road network
sensors. We can check the road network speed in real-time using Google Map. In other words,
the speed of a specific sensor is not sensitive and can be acquired by public service. Currently,
the Google Map example demonstrates that the road network sensors can be uploaded to
servers from transportation agencies without any privacy concerns.
(3) Another serious problem is that the DNN-based model (encoder-decoder used in the
proposed method) are not deployable in the sensor device due to computational resource
constraint. In practice, the number of road network sensor in transportation is huge, so it is
impractical to ask for an expensive upgrading which requires to install a powerful DNN enabled
chips.
***2. No technical contribution***
The focus of this work is diverse (both model and training), but lack of contribution in each
aspect.
(1) The authors argue the contribution of modeling (e.g., 4.1 modeling spatial dependencies, 4.2
“inductive v.s. transductive”). However, for the modeling part, I believe the contribution is from
GCN+GRU-based modeling, which is already there in many data mining publications (Equation
(1)(2)(3) are not newly proposed models). Please refer to [1][2]. The model used in this paper is
a simplified version of these two GCN+RNN-based models in transportation. More advanced
models following these two can obtain better performance than the model used in this paper.
[1] Spatio-Temporal Graph Convolutional Networks: A Deep Learning Framework for Traffic
Forecasting
[2] Diffusion convolutional recurrent neural network: Data-driven traffic forecasting
(2) The authors also claim the benefit of alternating optimization (4.3, 4.4). It is good to know
such an alternating method works well in practice. However, the author does not provide any
in-depth analysis or comprehensive experimental evidence (see my comments 3 below for
details).
In short, I suggest the author focuses on one aspect of federated learning to consolidate the
contribution. At least, the current version raises my concern that this work is a simple
combination of many techniques without meta-contribution in each one. If the overall
characteristics are good, it should be encouraged. However, the effectiveness is not obvious:
higher accuracy, security, fairness, privacy, communication, and computation efficiency, I
cannot see any of these advantages of such a combination:
[Model Performance] In terms of higher accuracy, as mentioned in (2), the model used is not
novel. More advanced models can obtain higher accuracy. See following works of 2-(2) [1][2] for
details.
[Optimization] The experimental results of the alternating optimization has some
counter-intuition results. See 3 for details.
[Security/Privacy] As for security and privacy, the hidden vector exchange may leak privacy but
this work does not discuss it. See 8 for details.
[Computational Efficiency] The model used in edge devices are too computational expensive for
resource constrained sensors. See 4 for details.
***3. The training Algorithm is novel but the experimental result is not convincing***
The training method is relatively novel in federated learning. However, the authors should either
provide enough intuition to explain why this work (any related works in optimization theory?) or
demonstrate it with experimental design. The author attempts to prove that alternating training
works better but it’s not convincing. In Section 4.3, all the experimental results are not
convincing or counterintuitive:
First, I am surprised that the authors have a result to say that split learning performs worse than
centralized training. In the essence of optimization, split learning is the same as centralized
training. The only difference is that split learning offloads part of the model architecture to the
server-side to reduce the communication/computation cost.
Second, due to the first reason, that “AT w/o FedAvg” and “AT + FedAvg” works better than SL
can NOT demonstrate the effectiveness of alternating optimization. As I mentioned previously,
to prove the effectiveness of alternating optimization, we need more analysis or comprehensive
experimental design.
Third, another counterintuitive result is that AT + FedAvg performs better than centralized
training (the red curve is lower than the blue curve). Normally, in federated learning, one should
believe that centralized training is the lower bound of training/validation errors. An alternating
local SGD method normally leads to variance bias in the training process, thus I believe it’s
impossible to obtain a better performance than centralized training.
The last concern is that the experiments only perform at one dataset (METR-LA). Please also
check whether the same conclusion holds in another dataset.
***4. ​Communication and computational cost***
The computational cost is evaluated, but what’s the implication of the number showing in terms
of FLOPS? I don’t think the resource constrained road network sensor can handle so many
FLOPS. Does any running time result in a real-world sensor device? If testing the sensor is
prohibited, please provide the running time result at a low-performance smartphone or other IoT
devices (NVIDIA edge GPU devices).
***5. ​Overall training time***
The overall training time and the bandwidth of exchanging hidden vectors should be mentioned.
If the network size of the exchange information is too high and the bandwidth is limited in
wireless communication (e.g., road sensor networks), a single iteration will cost too much time
or failure due to communication protocol constraint (e.g., in IoT setting, maximum payload
length is only 65535 bytes), which will lead to a long training time or impractical assumption.
Please discuss the size of exchange information (hidden vector/gradient) and the total training
time in revision.
***6. Scalability issue is ignored***
The proposed method does not use a client sampling strategy, a common practice in
cross-device FL (see the original FedAvg paper [1]), to mitigate the scalability issue. What the
performance if we want to learn 10 thousand sensors? Especially in the transportation setting,
the scalability is very important. However, the authors do not discuss this.
[1] Communication-Efficient Learning of Deep Networks from Decentralized Data.
https://arxiv.org/abs/1602.05629
***7. Dataset: Non-I.I.D. is not properly discussed***
A key assumption of Federated Learning is that datasets across devices are non-I.I.D. This is
largely ignored by the proposed methods. Slightly mentioning the heterogeneous property is not
enough. Please discuss the details of this in revision (e.g., show the distribution diagram). More
significantly, DNN models like encoder-decoder architectures normally eat a lot of samples, but
the number of samples in each node is small in practice. The assumption that each node has
enough dataset to train a good model is too strong (we cannot assume the edge device has the
storage capability to store months of time series data since the storage ability is limited at the
edge, ***4-6 months*** as the author mentioned). Maybe the alternating optimization method
helps, but it lacks analysis. In my opinion, it’s more interesting and practical to construct a
graph-level federated learning method since multiple nodes can be centralized to an edge
server belongs to a company or an agency like the transportation scenario.
***8. ​Privacy***
The authors discuss some privacy-preserving methods in related works, but the proposed
method does not include any privacy design. For example, [1] analyze potential leakage from
the hidden vectors, which proves that the proposed method does not have privacy advantages
than exchanging gradient/models.
[1] Cronus: Robust and Heterogeneous Collaborative Learning with Black-Box Knowledge
Transfer. https://arxiv.org/abs/1912.11279
***9. Lack of Related works***
Since alternating optimization (training algorithm) is the novel part of the proposed method,
there is a lack of literature review of this method, either in the general ML optimization literature,
or the specific federated learning publications. Without such a comparison, the novelty is not
convincing.
***10. Some misunderstandings of federated learning***
(1) In the first paragraph of the introduction, the authors claim that decentralized training can
improve latency, but this argument is vague. Improving training latency or inference latency?
Federated learning will cost time to train on devices, which requires a long time due to many
rounds of communication synchronization. For the inference, to improve the latency, we need
model compression techniques, which is also not the main goal of the proposed method.
(2) The term “decentralized training” is misused. In federated learning, decentralized training
also refers to training using a decentralized topology (no central server). Note that FedAvg used
in the proposed method is a centralized training with decentralized datasets. Please clarify the
difference in revision, making sure the readers understand which aspect is decentralized, the
datasets, the distributed optimization algorithm, or just computation.
***11. Reproducibility***
The proposed algorithm is complex. Without the code release, it’s hard to check the correctness
of the implementation. Deep learning normally costs time, I am curious to know how the authors
implement such a complex system/algorithm which can train so many nodes in parallel. If it is a
simulator, the training time will be too long since so there so many nodes (300+) with DNN
models. If the authors can provide code, maybe all my confusion can be addressed, and I am
willing to increase my rating.
***Suggestions***
(1) I encourage the authors demonstrate the idea with more realistic datasets. For example,
in social network, it is possible to run models on the smartphones of Internet users to
protect their privacy, which is much better than the transportation example.
(2) In the current version, there is no contribution in modeling. Thus, I suggest authors to
think about customizing the model to obtain benefit of efficiency and privacy.
(3) More intuition and discussion are requires for the training method. Why alternating
minimization works well in the GCN setting.
(4) Since the training results are not convincing, I suggest authors run more experiments to
understand the proposed method and explain why there are some counter-intuitive
results.
(5) Privacy is not the main focus, but some discussions and related works should be
mentioned.
Overall Comments
Due to so many concerns mentioned above, I encourage the authors to have a deeper
understanding of federated learning and address these issues in the revision, especially the
motivation, and demonstrating the effectiveness of the proposed alternating training method.
After thinking a while, I cannot find a practical scenario that requires GCN-based node-level
privacy. At least, the sensor dataset is not a good example. Maybe the authors can search for a
practical dataset to demonstrate motivation.
In addition, please focus on just one gist rather than intertwine too many aspects and claim all
contributions without any trade-off discussion. Besides the training framework, it’s also
interesting research if we can see newly developed model architectures can trade-off many
important aspects in FL.
review_title: To broaden the application of federated learning, the author proposed a node-level distributed training framework for an existing spatial-temporal model (GCN + GRU), which aims to protect the privacy or circumvent the difficulty of centralizing datasets.

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes a new modeling framework which would utilize the spatio-temporal information in the data by developing a Cross-Node-Federated-Graph-Neural-Network (CNFGNN). The methodology seems to be clearly explained while it relies on many existing methods in the literature which limits the novelty of the paper. The model is applied to two real data sets and outperforms some existing methods. Overall, I find the methodology interesting.

Some comments:

In the traffic flow forecasting data: (1) Have you tried different weights (W_{i,j}) instead of the Gaussian kernel function? Does the performance measurements change by using a different weight function? It would be appropriate to show the robustness of the method with respect to changes in the spatial weight selection. (2) Removing trend and seasonality is a common practice in time series forecasting, but it is not clear whether this step is addressed properly or not. What about the seasonality in the data? Is this issue resolved in the preprocessing? 
review_title: Network modeling using Cross-Node-Federated-Graph-Neural-Network (CNFGNN)

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper presents an algorithm to model the complex spatio-temporal dependencies across multiple nodes under a federated learning setting. Specifically, the authors employ a GRU-based encoder-decoder architecture to construct the node-level temporal dynamics on each node and utilize a GCN to capture server-level spatial dynamics on the server. 

The model consists of four major steps. First, each node encodes the local data and sends the extracted temporal vectors to the server. The server then collects all temporal vectors, builds the graph, extracts the spatial vectors, and broadcasts the server-level dynamics. Next, each node combines the node-level temporal dynamics and the server-level spatial dynamics to decode the prediction.  Finally, the backpropagation algorithm is performed. 

Pros -------------

- It is interesting to extend the concept of federated learning to graph-based spatio-temporal forecasting tasks. 
- The paper is well-written and structured.
- Split learning seems to be a good idea to reduce communication costs.

Cons  -------------

- Privacy concern: This paper introduces some privacy-preserving methods in Related Work, but the corresponding protection methods do not present in the paper. Or the author regards the extracted feature vectors and corresponding gradients are not sensitive. 

- Communication costs: Although the model makes several adjustments, the amount of data transmitted in a complete training phase is still very large (200+GB), which places a demanding requirement on the capability of edge devices. In addition, I am curious about where communication costs mainly come from. the vectors and gradients from SplitNN? or the model parameters from FedAvg?

- Experiments:  An important baseline is missing, a model that aggregates all the data and directly combines GRU with GNN to predict the traffic flow without a federated setting (like DCRNN). In my view, this model should be the upper bound of the model performance, just like what the author did in section 4.3-(1). Besides, there is no explanation for the little change in the performance of GRU+FedAvg in section 4.2, what if we use more data like 90% or fewer data like 5%. Besides, the main difference between GRU+FedAvg and CNFGNN is whether to use GNN to capture spatial dependencies.  Is it spatial dependencies as the key reason that achieves better performances in an inductive learning setting?
review_title: This paper extends federated learning to graph-based spatio-temporal forecasting tasks. However, privacy issues, communication cost issues, and experiments need to be improved and clarified.

Meta-review: This paper received mixed reviews: two positives (6, 6) and two negatives (5, 3). However, the positive reviewers have very low confidence, do not show strong supports for this paper. The reviewers raised various concerns about this paper, and there still exist remaining critical issues although the authors made substantial efforts to answer the questions.

After reading the paper and all the comments by the reviewers, I decided to recommend rejecting this paper mainly due to its weak technical contribution and ignorance of privacy issues. Note that this opinion is shared with two negative reviewers. The proposed model and alternative training scheme are straightforward, and the novelty is not distinct. Also, the authors seem to assume that "the extracted feature vectors and corresponding gradients are not sensitive". This comment is given by R2 but has not been clarified. The proposed method is lacking in this aspect and it is hard to say that it is an FL approach.


Decision: Reject