Federated Graph Classification over Non-IID Graphs

1 Introduction . Federated learning ( FL ) as a distributed learning paradigm that trains centralized models on decentralized data has attracted much attention recently [ 28 , 53 , 25 , 18 , 17 ] . FL allows local systems to benefit from each other while keeping their own data private . Especially , for local systems with scarce training data or lack of diverse distributions , FL provides them with the potentiality to leverage the power of data from others , in order to facilitate the performance on their own local tasks . One important problem FL concerns is data distribution heterogeneity , since the decentralized data , collected by different institutes using different methods and aiming at different tasks , are highly likely to follow non-identical distributions . Prior works approach this problem from different aspects , including optimization process [ 25 , 18 ] , personalized FL [ 13 , 6 , 8 ] , clustered FL [ 9 , 15 , 2 ] , etc . As more advanced techniques are developed for learning with graph data , using graphs to model and solve real-world problems becomes more popular . One important scenario of graph learning is graph classification , where models such as graph kernels [ 44 , 34 , 36 , 45 ] and graph neural networks [ 21 , 43 , 49 , 46 , 47 , 48 ] are used to predict graph-level labels based on the features and structures of graphs . One real scenario of graph classification is molecular property prediction , which is an important task in cheminformatics and AI medicine . In the area of bioinformatics , graph classification can be used to learn the representation of proteins and classify them into enzymes or non-enzymes . For collaboration networks , sub-networks can be classified regarding the information of research areas , topics , genre , etc . More applicable scenarios include geographic networks , temporal networks , etc . ⇤Corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Since the key idea of FL is the sharing of underlying common information , as [ 23 ] discusses that real-world graphs preserve many common properties , we become curious about the question , whether real-world graphs from heterogeneous sources ( e.g. , different datasets or even divergent domains ) can provide useful common information among each other ? To understand this question , we first conduct preliminary data analysis to explore real-world graph properties , and try to find clues about common patterns shared among graphs across datasets . As shown in Table 1 , we analyze four typical datasets from different domains , i.e. , PTC_MR ( molecular structures ) , ENZYMES ( protein structures ) , IMDB-BINARY ( social communities ) , and MSRC_21 ( superpixel networks ) . We find them to indeed share certain properties that are statistically significant compared to random graphs with the same numbers of nodes and links ( generated with the Erdős–Rényi model [ 7 , 10 ] ) . Such observations confirm the claim about common patterns underlying real-world graphs , which can largely influence the graph mining models and motivates us to consider the FL of graph classification across datasets and even domains . More details and discussion about Table 1 can be found in Appendix A . Although common patterns exist among graph datasets , we can still observe certain heterogeneity . In fact , the detailed graph structure distributions and node feature distributions can both diverge due to various reasons . To demonstrate this , we design and evaluate a structure heterogeneity measure and a feature heterogeneity measure in different scenarios ( c.f . Section 4.1 ) . We refer to the graphs possibly with significant heterogeneity in our cross-dataset FL setting as non-IID graphs , which concerns both structure non-IID and feature non-IID , where naïve FL algorithms like FedAvg [ 28 ] can fail and even backfire ( c.f . Section 6.2 ) . Moreover , as the heterogeneity varies from case to case , a dynamic FL algorithm is needed to keep track of such heterogeneity of non-IID graphs while conducting collaborative model training . Due to the observations that the graphs in one client can be similar to those in some clients but not the others , we get motivated by [ 2 ] and find it intuitive to consider a clustered FL framework , which assigns local clients to multiple clusters with less data heterogeneity . To this end , we propose a novel graph-level clustered FL framework ( termed GCFL ) through integrating the powerful graph neural networks ( GNNs ) such as GIN [ 43 ] into clustered FL , where the server can dynamically cluster the clients based on the gradients of GNN without additional prior knowledge , while collaboratively training multiple GNNs as necessary for homogeneous clusters of clients . We theoretically analyze that the model parameters of GNN indeed reflect the structures and features of graphs , and thus using the gradients of GNN for clustering in principle can yield clusters with reduced heterogeneity of both structures and features . In addition , we conduct empirical analysis to support the motivation of clustered FL across heterogeneous graph datasets in Appendix A . Although GCFL can theoretically achieve homogeneous clusters , during its training , we observe that the gradients transmitted at each communication round fluctuate a lot ( c.f . Section 5.1 ) , which could be caused by the complicated interactions among clients regarding both structure and feature heterogeneity , making the local gradients towards divergent directions . In the vanilla GCFL framework , the server calculates a matrix for clustering only based on the last transmitted gradients , which ignores the client ’ s multi-round behaviors . Therefore , we further propose an improved version of GCFL with gradient-series-based clustering ( termed GCFL+ ) . We conduct extensive experiments with various settings to demonstrate the effectiveness of our frameworks . Moreover , we provide in-depth analysis on the capability of them on reducing both structure and feature heterogeneity of clients through clustering . Lastly , we analyze the convergence of our frameworks . The experimental results show surprisingly positive results brought by our novel setting of cross-dataset/cross-domain FL for graph classification , where our GCFL+ framework can effectively and consistently outperform other straightforward baselines . 2 Related works . Federated Learning Federated learning ( FL ) has gained increasing attention as a training paradigm under the setting where data are distributed at remote devices and models are collaboratively trained under the coordination of a central server . FedAvg was first proposed by [ 27 ] which illustrates the general setting of an FL framework . Since the original FedAvg relies on the optimization by SGD , data of non-IID distribution will not guarantee the stochastic gradients to be an unbiased estimation of the full gradients , thus hurting the convergence of FL . In fact , multiple experiments [ 53 , 25 , 18 ] have shown that the convergence will be slow and unstable , and the accuracy will degrade with FedAvg when data at each client are statistically heterogeneous ( non-IID ) . [ 53 , 15 , 12 ] proposed different data sharing strategies to tackle the data heterogeneity problem by sharing the local device data or serverside proxy data , which still requires certain public common data , whereas other studies explored the convergence guarantee under the non-IID setting by assuming bounded gradients [ 39 , 51 ] or additional noise [ 19 ] . There are also works seeking to reduce the variance of the clients [ 26 , 18 , 25 ] . Furthermore , multiple works have been proposed to explore the connection between model-agnostic meta-learning ( MAML ) and personalized federated learning [ 8 , 4 ] . They aim to learn a generalizable global model and then fine-tune it on local clients , which may still fail when data on local clients are from divergent domains with high heterogeneity . Some personalized FL works [ 6 , 24 ] studied the bi-level problem of optimization which decouples the local and global optimization , but having each client maintain its own model can lead to high communication cost among the clients and the server . While personalization in the FL setting can address the client heterogeneity problem to some extent , the clustered FL framework [ 33 ] can incorporate personalization at the group level to keep the benefits of personalized FL and reduce the communication cost simultaneously . Federated Learning on Graphs Although FL has been intensively studied with Euclidean data such as images , there exist few studies about FL for graph data . [ 22 ] first introduced FL on graph data , by regarding each client as a node in a graph . [ 3 ] studied the cross-domain heterogeneity problem in FL by leveraging Graph Convolutional Networks ( GCNs ) to model the interaction between domains . [ 29 ] studied spatio-temporal data modeling in the FL setting by leveraging a Graph Neural Network ( GNN ) based model to capture the spatial relationship among clients . [ 5 ] proposed a generalized federated knowledge graph embedding framework that can be applied for multiple knowledge graph embedding algorithms . Moreover , there are several works exploring the GNNs under the FL setting : [ 16 , 54 , 40 ] focused on the privacy issue of federated GNNs ; [ 37 ] incorporated model-agnostic metalearning ( MAML ) into graph FL , which handled non-IID graph data while also preserving the model generalizability ; [ 52 ] studied the missing neighbor generation problem in the subgraph FL setting ; [ 38 ] proposed a computationally efficient way of GCN architecture search with FL ; [ 11 ] implemented an open FL benchmark system for GNNs . Most existing works consider node classification and link prediction on graphs , which can not be trivially applied to our graph classification setting . 3 Preliminaries . 3.1 Graph Neural Networks ( GNNs ) . [ 42 ] provides a taxonomy that categorizes Graph Neural Networks ( GNNs ) into recurrent GNNs , convolutional GNNs , graph autoencoders , and spatial-temporal GNNs . In general , given the structure and feature information of a graph G = ( V , E , X ) , where V , E , X denote nodes , links and node features , GNNs target to learn the representations of graphs , such as a node embedding hv 2 Rdv , or a graph embedding hG 2 RdG . A GNN typically consists of message propagation and neighborhood aggregation , in which each node iteratively gathers the information propagated by its neighbors , and aggregates them with its own information to update its representation . Generally , an L-layer GNN can be formulated as h ( l+1 ) v = ( h ( l ) v , agg ( { h ( l ) u ; u 2 Nv } ) ) , 8l 2 [ L ] , ( 1 ) where h ( l ) v is the representation of node v at the lth layer , and h ( 0 ) v = xv is the node feature . Nv is neighbors of node v , agg ( · ) is an aggregation function that can vary for different GNN variants , and represents an activation function . For a graph-level representation hG , it can be pooled from the representations of all nodes , as hG = readout ( { hv ; v 2 V } ) , ( 2 ) where readout ( · ) can be implemented as mean pooling , sum pooling , etc , which essentially aggregates the embeddings of all nodes on the graph into a single embedding vector to achieve tasks like graph classification and regression . 3.2 The FedAvg algorithm McMahan et al . [ 28 ] proposed an SGD-based aggregating algorithm , FedAvg , based on the fact that SGD is widely used and powerful for optimization . FedAvg is the first basic FL algorithm and is commonly used as the starting point for more advance FL framework design . The key idea of FedAvg is to aggregate the updated model parameters transmitted from local clients and then re-distribute the averaged parameters back to each client . Specifically , given m clients in total , at each communication round t , the server first samples a partition of clients { Si } ( t ) . For each client Si in { Si } ( t ) , it trains the model downloaded from the server locally with its own data distribution Di for Elocal epochs . The client Si then transmits its updated parameters w ( t ) i to the server , and the server will aggregate these updates by w ( t+1 ) = mX i=1 |Di| |D| w ( t ) i , ( 3 ) where |Di| is the size of data samples in client Si and |D| is the total size of samples over all clients . After generating the aggregated parameters ( the global model updates ) , the server broadcasts the new parameters w ( t+1 ) to remote clients , and at the ( t+ 1 ) round clients use w ( t+1 ) to start their local training for another Elocal epochs . 4 The GCFL framework

Summary: This paper provides a timely study on the emerging domain of graph federated learning, focusing on the graph classification setting. This is a natural setting close to federated learning over other types of data in terms of data split, yet the authors manage to motivate the need of doing so based on real network data analysis, and figure out the unique challenges of graphs regarding the non-iid nature of both structures and features. This provides a good motivation to the usage of GNNs, which is supported by some theoretical analysis. The existing technique of clustered FL is adopted to the GNNs and slightly improved for better performance.