Federated Graph Classification over Non-IID Graphs

1 Introduction . Federated learning ( FL ) as a distributed learning paradigm that trains centralized models on decentralized data has attracted much attention recently [ 28 , 53 , 25 , 18 , 17 ] . FL allows local systems to benefit from each other while keeping their own data private . Especially , for local systems with scarce training data or lack of diverse distributions , FL provides them with the potentiality to leverage the power of data from others , in order to facilitate the performance on their own local tasks . One important problem FL concerns is data distribution heterogeneity , since the decentralized data , collected by different institutes using different methods and aiming at different tasks , are highly likely to follow non-identical distributions . Prior works approach this problem from different aspects , including optimization process [ 25 , 18 ] , personalized FL [ 13 , 6 , 8 ] , clustered FL [ 9 , 15 , 2 ] , etc . As more advanced techniques are developed for learning with graph data , using graphs to model and solve real-world problems becomes more popular . One important scenario of graph learning is graph classification , where models such as graph kernels [ 44 , 34 , 36 , 45 ] and graph neural networks [ 21 , 43 , 49 , 46 , 47 , 48 ] are used to predict graph-level labels based on the features and structures of graphs . One real scenario of graph classification is molecular property prediction , which is an important task in cheminformatics and AI medicine . In the area of bioinformatics , graph classification can be used to learn the representation of proteins and classify them into enzymes or non-enzymes . For collaboration networks , sub-networks can be classified regarding the information of research areas , topics , genre , etc . More applicable scenarios include geographic networks , temporal networks , etc . ⇤Corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Since the key idea of FL is the sharing of underlying common information , as [ 23 ] discusses that real-world graphs preserve many common properties , we become curious about the question , whether real-world graphs from heterogeneous sources ( e.g. , different datasets or even divergent domains ) can provide useful common information among each other ? To understand this question , we first conduct preliminary data analysis to explore real-world graph properties , and try to find clues about common patterns shared among graphs across datasets . As shown in Table 1 , we analyze four typical datasets from different domains , i.e. , PTC_MR ( molecular structures ) , ENZYMES ( protein structures ) , IMDB-BINARY ( social communities ) , and MSRC_21 ( superpixel networks ) . We find them to indeed share certain properties that are statistically significant compared to random graphs with the same numbers of nodes and links ( generated with the Erdős–Rényi model [ 7 , 10 ] ) . Such observations confirm the claim about common patterns underlying real-world graphs , which can largely influence the graph mining models and motivates us to consider the FL of graph classification across datasets and even domains . More details and discussion about Table 1 can be found in Appendix A . Although common patterns exist among graph datasets , we can still observe certain heterogeneity . In fact , the detailed graph structure distributions and node feature distributions can both diverge due to various reasons . To demonstrate this , we design and evaluate a structure heterogeneity measure and a feature heterogeneity measure in different scenarios ( c.f . Section 4.1 ) . We refer to the graphs possibly with significant heterogeneity in our cross-dataset FL setting as non-IID graphs , which concerns both structure non-IID and feature non-IID , where naïve FL algorithms like FedAvg [ 28 ] can fail and even backfire ( c.f . Section 6.2 ) . Moreover , as the heterogeneity varies from case to case , a dynamic FL algorithm is needed to keep track of such heterogeneity of non-IID graphs while conducting collaborative model training . Due to the observations that the graphs in one client can be similar to those in some clients but not the others , we get motivated by [ 2 ] and find it intuitive to consider a clustered FL framework , which assigns local clients to multiple clusters with less data heterogeneity . To this end , we propose a novel graph-level clustered FL framework ( termed GCFL ) through integrating the powerful graph neural networks ( GNNs ) such as GIN [ 43 ] into clustered FL , where the server can dynamically cluster the clients based on the gradients of GNN without additional prior knowledge , while collaboratively training multiple GNNs as necessary for homogeneous clusters of clients . We theoretically analyze that the model parameters of GNN indeed reflect the structures and features of graphs , and thus using the gradients of GNN for clustering in principle can yield clusters with reduced heterogeneity of both structures and features . In addition , we conduct empirical analysis to support the motivation of clustered FL across heterogeneous graph datasets in Appendix A . Although GCFL can theoretically achieve homogeneous clusters , during its training , we observe that the gradients transmitted at each communication round fluctuate a lot ( c.f . Section 5.1 ) , which could be caused by the complicated interactions among clients regarding both structure and feature heterogeneity , making the local gradients towards divergent directions . In the vanilla GCFL framework , the server calculates a matrix for clustering only based on the last transmitted gradients , which ignores the client ’ s multi-round behaviors . Therefore , we further propose an improved version of GCFL with gradient-series-based clustering ( termed GCFL+ ) . We conduct extensive experiments with various settings to demonstrate the effectiveness of our frameworks . Moreover , we provide in-depth analysis on the capability of them on reducing both structure and feature heterogeneity of clients through clustering . Lastly , we analyze the convergence of our frameworks . The experimental results show surprisingly positive results brought by our novel setting of cross-dataset/cross-domain FL for graph classification , where our GCFL+ framework can effectively and consistently outperform other straightforward baselines . 2 Related works . Federated Learning Federated learning ( FL ) has gained increasing attention as a training paradigm under the setting where data are distributed at remote devices and models are collaboratively trained under the coordination of a central server . FedAvg was first proposed by [ 27 ] which illustrates the general setting of an FL framework . Since the original FedAvg relies on the optimization by SGD , data of non-IID distribution will not guarantee the stochastic gradients to be an unbiased estimation of the full gradients , thus hurting the convergence of FL . In fact , multiple experiments [ 53 , 25 , 18 ] have shown that the convergence will be slow and unstable , and the accuracy will degrade with FedAvg when data at each client are statistically heterogeneous ( non-IID ) . [ 53 , 15 , 12 ] proposed different data sharing strategies to tackle the data heterogeneity problem by sharing the local device data or serverside proxy data , which still requires certain public common data , whereas other studies explored the convergence guarantee under the non-IID setting by assuming bounded gradients [ 39 , 51 ] or additional noise [ 19 ] . There are also works seeking to reduce the variance of the clients [ 26 , 18 , 25 ] . Furthermore , multiple works have been proposed to explore the connection between model-agnostic meta-learning ( MAML ) and personalized federated learning [ 8 , 4 ] . They aim to learn a generalizable global model and then fine-tune it on local clients , which may still fail when data on local clients are from divergent domains with high heterogeneity . Some personalized FL works [ 6 , 24 ] studied the bi-level problem of optimization which decouples the local and global optimization , but having each client maintain its own model can lead to high communication cost among the clients and the server . While personalization in the FL setting can address the client heterogeneity problem to some extent , the clustered FL framework [ 33 ] can incorporate personalization at the group level to keep the benefits of personalized FL and reduce the communication cost simultaneously . Federated Learning on Graphs Although FL has been intensively studied with Euclidean data such as images , there exist few studies about FL for graph data . [ 22 ] first introduced FL on graph data , by regarding each client as a node in a graph . [ 3 ] studied the cross-domain heterogeneity problem in FL by leveraging Graph Convolutional Networks ( GCNs ) to model the interaction between domains . [ 29 ] studied spatio-temporal data modeling in the FL setting by leveraging a Graph Neural Network ( GNN ) based model to capture the spatial relationship among clients . [ 5 ] proposed a generalized federated knowledge graph embedding framework that can be applied for multiple knowledge graph embedding algorithms . Moreover , there are several works exploring the GNNs under the FL setting : [ 16 , 54 , 40 ] focused on the privacy issue of federated GNNs ; [ 37 ] incorporated model-agnostic metalearning ( MAML ) into graph FL , which handled non-IID graph data while also preserving the model generalizability ; [ 52 ] studied the missing neighbor generation problem in the subgraph FL setting ; [ 38 ] proposed a computationally efficient way of GCN architecture search with FL ; [ 11 ] implemented an open FL benchmark system for GNNs . Most existing works consider node classification and link prediction on graphs , which can not be trivially applied to our graph classification setting . 3 Preliminaries . 3.1 Graph Neural Networks ( GNNs ) . [ 42 ] provides a taxonomy that categorizes Graph Neural Networks ( GNNs ) into recurrent GNNs , convolutional GNNs , graph autoencoders , and spatial-temporal GNNs . In general , given the structure and feature information of a graph G = ( V , E , X ) , where V , E , X denote nodes , links and node features , GNNs target to learn the representations of graphs , such as a node embedding hv 2 Rdv , or a graph embedding hG 2 RdG . A GNN typically consists of message propagation and neighborhood aggregation , in which each node iteratively gathers the information propagated by its neighbors , and aggregates them with its own information to update its representation . Generally , an L-layer GNN can be formulated as h ( l+1 ) v = ( h ( l ) v , agg ( { h ( l ) u ; u 2 Nv } ) ) , 8l 2 [ L ] , ( 1 ) where h ( l ) v is the representation of node v at the lth layer , and h ( 0 ) v = xv is the node feature . Nv is neighbors of node v , agg ( · ) is an aggregation function that can vary for different GNN variants , and represents an activation function . For a graph-level representation hG , it can be pooled from the representations of all nodes , as hG = readout ( { hv ; v 2 V } ) , ( 2 ) where readout ( · ) can be implemented as mean pooling , sum pooling , etc , which essentially aggregates the embeddings of all nodes on the graph into a single embedding vector to achieve tasks like graph classification and regression . 3.2 The FedAvg algorithm McMahan et al . [ 28 ] proposed an SGD-based aggregating algorithm , FedAvg , based on the fact that SGD is widely used and powerful for optimization . FedAvg is the first basic FL algorithm and is commonly used as the starting point for more advance FL framework design . The key idea of FedAvg is to aggregate the updated model parameters transmitted from local clients and then re-distribute the averaged parameters back to each client . Specifically , given m clients in total , at each communication round t , the server first samples a partition of clients { Si } ( t ) . For each client Si in { Si } ( t ) , it trains the model downloaded from the server locally with its own data distribution Di for Elocal epochs . The client Si then transmits its updated parameters w ( t ) i to the server , and the server will aggregate these updates by w ( t+1 ) = mX i=1 |Di| |D| w ( t ) i , ( 3 ) where |Di| is the size of data samples in client Si and |D| is the total size of samples over all clients . After generating the aggregated parameters ( the global model updates ) , the server broadcasts the new parameters w ( t+1 ) to remote clients , and at the ( t+ 1 ) round clients use w ( t+1 ) to start their local training for another Elocal epochs . 4 The GCFL framework

Summary: The paper proposes a federated graph classification system based on clustered federated learning to solve the non-iid problem in graph structure and node features across clients. The paper also proposes to use dynamic time warping to improve clustering performance. The authors evaluate their system on a couple of different scenarios: (i) a single graph dataset split across multiple clients and (ii) Graph datasets from different domains on each client.