Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper provides a timely study on the emerging domain of graph federated learning, focusing on the graph classification setting. This is a natural setting close to federated learning over other types of data in terms of data split, yet the authors manage to motivate the need of doing so based on real network data analysis, and figure out the unique challenges of graphs regarding the non-iid nature of both structures and features. This provides a good motivation to the usage of GNNs, which is supported by some theoretical analysis. The existing technique of clustered FL is adopted to the GNNs and slightly improved for better performance. 
main_review: This paper does not try to sell a bunch of fancy or complicated architectures. Instead, it tells a real scientific story, where every major step including the problem setting, usage of GNN, utilization of clustered FL (and the small improvement) are well motivated and justified through real data analysis and observations. This has been taken for granted in recent works especially related to graph federated learning. In terms of weaknesses, I find certain low-level technical details to be confusing, such as why agglomerative clustering is used for splitting the clusters? Is it following the design of existing clustered FL ?  Is it really ideal?

Overall, this paper is well-organized and easy to follow. It provides enough information even for non-expert readers.

This paper properly justifies the need of graph classification with federated learning and demonstrates its benefit in different settings (one-dataset, cross-dataset, cross-domain). An easy-to-implement framework composed of GNN and clustered FL is proposed and shown to be effective. Though there exist some issues such as model efficiency, privacy, etc. that should be further studied, this work can serve as a good starting point for research on graph federated learning.

limitations_and_societal_impact: See main review
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper advocates a novel setting of cross-dataset and cross-domain federated learning for graph classification, which allows multiple data owners with graphs of non-iid structures and features to collaboratively train powerful graph classifiers without direct data sharing. The proposed techniques are neat yet well-justified. Extensive experiment results show clear performance gains of the proposed GCFL+ over vanilla baselines like FedAvg across different settings. Convergence analyses are also provided.
main_review: Some minor concerns are as follows:

Q1: It looks like the objective of GCFL is closer to personalized federated learning instead of standard federated learning. Can the authors clarify on this?

Q2: Although simplicity is good, the overall technical novelty is somewhat limited. For example, how is the proposed framework different from recent works such as FedCG (Cluster-driven Graph Federated Learning over Multiple Domains), in CVPR’21 workshop?

Q3: The columns in tables 3&4 can use more explanations, such as the min gain and ratio columns. As they appear now, it requires more thinking to exactly appreciate how they show the advantages of the proposed methods.

Q4: How do hyper-parameters like \epsilon_1 and \epsilon_2 influence the performance of the framework?


limitations_and_societal_impact: Please see Main Review.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The paper proposes a federated graph classification system based on clustered federated learning to solve the non-iid problem in graph structure and node features across clients. The paper also proposes to use dynamic time warping to improve clustering performance. The authors evaluate their system on a couple of different scenarios: (i) a single graph dataset split across multiple clients and (ii) Graph datasets from different domains on each client. 

main_review: Summary 

The paper proposes a federated graph classification system based on clustered federated learning to solve the non-iid problem in graph structure and node features across clients. The paper also proposes to use dynamic time warping to improve clustering performance. The authors evaluate their system on a couple of different scenarios: (i) a single graph dataset split across multiple clients and (ii) Graph datasets from different domains on each client. 

Pros

The authors recognize an important problem in the non-iidness that occurs in graph datasets within the same domain and naturally across domains. The paper is mostly well written and easy to follow, although a few sections could benefit from clear phrasing. 

Cons

The main concern with the paper are a lack of technical novelty, an unconvincing motivation, and inadequate experimental evaluation. Detailed comments are given below.


Detailed comments:
It's unclear what the comparison of random graphs to real world graphs is meant to illustrate beyond the fact that real world graphs illustrate certain common properties across domains. This is a fairly well understood phenomenon and it seemed there was not much insight with this data in the introduction.
 
The motivation for clustering is entirely dependent on a robust metric of homogeneity. On the one hand, statistical structural properties (as shown in Table 1) might be similar between graphs on different clients. But how much do these statistical properties matter? What is the correlation between the expressivity of a GNN trained on different graphs and the statistical similarity between those graphs? 


GCFL
"From Table 1 we notice that real-world graphs tend to share certain general properties across different graphs, datasets and even domains..". This is probably a bit strong claim to be made from the Table 1. The information presented in Table 1 is inadequate to make this claim. At the very least more datasets need to be investigated.  Table 1 in itself doesn't seem like a strong enough motivation for clustering. 

 Graph Clustered Federated Learning needs to be motivated with a real world use case that satisfies these properties. Where do real world scenarios with different clients containing heterogeneous features in their graphs / nodes show up? Why would an IMDB-BINARY type dataset ever be split between clients that cannot share data with each other? Or when would a client working on molecular data ever collaborate with a social media company working on social networks? Recent work https://arxiv.org/abs/2104.07145 motivated federated graph learning in the context of molecular graphs, but the features that are used in such a setting are often atom properties which would be fairly homogenous between graphs and clients. So the motivation needs to be better.

The technical novelty is limited. Section 4.3 seems more like an application of CFL than a novel approach to clustered federated learning. Moreover, the approach seems general and doesn't seem specific to exploiting any GNN related specifics. 

GCFL+
Observations on gradient norm and why GCFL cannot adjust itself makes sense, still a general phenomenon in FL.
Using DTW is simple and neat, but experimentally, improvement seems not significant. Have you ever tried different window lengths ? 

Experiments
In the single dataset to number of clients setting, how is the dataset partitioned in a non-iid fashion between clients? 

The baselines are not strong enough. What is the motivation for the self-train baseline? 

How does GCFL compare to just CFL being applied?

The claim that GCFL can help clients that are training on disparate graphs (different domains) collaborate needs to be better justified. Maybe I misunderstood, but it seems that  the authors are suggesting that the semantic information in the graphs on each client is  irrelevant? I am not sure if a client training on social network data can collaborate with a client training on some other  GCN models. Hence, collaboration across domains may not be that trivial, in my view. The experimentsalsection needs to be reinforced with  more combinations of datasets from different domains to convince readers  that such a non-intuitive result holds true in practice. 

Proofs
Why is the model used for analysis SGC but the model used in experiments GIN? I understand that it is for the sake of simplicity, but the models are quite different. Do the proofs hold up if the model being analyzed was GIN? Some intuition on this, even if not a full proof would definitely clear things up. 

limitations_and_societal_impact: No comments here to adress.
review_title: null

Meta-review: There was a fruitful discussion about this paper.
Overall, I feel that the paper contains enough interesting and novel ideas for a publication. 
Decision: Accept (Poster)