Subgraph Federated Learning with Missing Neighbor Generation

1 Introduction . Graph mining leverages links among connected nodes in graphs to conduct inference . Recently , graph neural networks ( GNNs ) have gained applause with impressing performance and generalizability in many graph mining tasks [ 29 , 11 , 16 , 20 , 32 ] . Similar to machine learning tasks in other domains , attaining a well-performed GNN model requires its training data to not only be sufficient but also follow the similar distribution as general queries . While in reality , data owners often collect limited and biased graphs and can not observe the global distribution . With heterogeneous subgraphs separately stored in local data owners , accomplishing a globally applicable GNN requires collaboration . Federated learning ( FL ) [ 17 , 35 ] , targeting at training machine learning models with data distributed in multiple local systems to resolve the information-silo problem , has shown its advantage in enhancing the performance and generalizability of the collaboratively trained models without the need of sharing any actual data . For example , FL has been devised in computer vision ( CV ) and natural language processing ( NLP ) to allow the joint training of powerful and generalizable deep convolutional neural networks and language models on separately stored datasets of images and texts [ 19 , 6 , 18 , 39 , 13 ] . Motivating Scenario . Taking the healthcare system as an example , as shown in Fig . 1 , residents of a city may go to different hospitals for various reasons . As a result , their healthcare data , such as demographics and living conditions , as well as patient interactions , such as co-staying in a sickroom and co-diagnosis of a disease , are stored only within the hospitals they visit . When any healthcare problem is to be studied in the whole city , e.g. , the prediction of infections when a pandemic occurs , ∗Corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . a single powerful graph mining model is needed to conduct effective inference over the entire global patient network , which contains all subgraphs from different hospitals . However , it is rather difficult to let all hospitals share their patient networks with others to train the graph mining model due to conflicts of interests and privacy concerns . In such scenarios , it is desirable to train a powerful and generalizable graph mining model over multiple distributed subgraphs without actual data sharing . However , this novel yet realistic setting brings two unique technical challenges , which have never been explored so far . Challenge 1 : How to jointly learn from multiple local subgraphs ? In our considered scenario , the global graph is distributed into a set of small subgraphs with heterogeneous feature and structure distributions . Training a separate graph mining model on each subgraph may not capture the global data distribution and is also prone to overfitting . Moreover , it is unclear how to integrate multiple graph mining models into a universally applicable one that can handle any queries from the underlying global graph . Solution 1 : FedSage : Training GraphSage with FedAvg . To attain a powerful and generalizable graph mining model from small and biased subgraphs distributed in multiple local owners , we develop a framework of subgraph federated learning , specifically , with the vanilla mechanism of FedAvg [ 21 ] . As for the graph mining model , we resort to GraphSage [ 11 ] , due to its advantages of inductiveness and scalability . We term this framework as FedSage . Challenge 2 : How to deal with missing links across local subgraphs ? Unlike distributed systems in other domains such as CV and NLP , whose data samples of images and texts are isolated and independent , data samples in graphs are connected and correlated . Most importantly , in a subgraph federated learning system , data samples in each subgraph can potentially have connections to those in other subgraphs . These connections carrying important information of node neighborhoods and serving as bridges among the data owners , however , are never directly captured by any data owner . Solution 2 : FedSage+ : Generating missing neighbors along FedSage . To deal with crosssubgraph missing links , we add a missing neighbor generator on top of FedSage and propose a novel FedSage+ model . Specifically , for each data owner , instead of training the GraphSage model on the original subgraph , it first mends the subgraph with generated cross-subgraph missing neighbors and then applies FedSage on the mended subgraph . To obtain the missing neighbor generator , each data owner impairs the subgraph by randomly holding out some nodes and related links and then trains the generator based on the held-out neighbors . Training the generator on an individual local subgraph enables it to generate potential missing links within the subgraph . Further training the generator in our subgraph FL setting allows it to generate missing neighbors across distributed subgraphs . We conduct experiments on four real-world datasets with different numbers of data owners to better simulate the application scenarios . According to our results , both of our models outperform locally trained classifiers in all scenarios . Compared to FedSage , FedSage+ further promotes the performance of the outcome classifier . Further in-depth model analysis shows the convergence and generalization ability of our frameworks , which is corroborated by our theoretical analysis in the end . 2 Related works . Graph mining . Graph mining emerges its significance in analyzing the informative graph data , which range from social networks to gene interaction networks [ 31 , 33 , 34 , 24 ] . One of the most frequently applied tasks on graph data is node classification . Recently , graph neural networks ( GNNs ) , e.g. , graph convolutional networks ( GCN ) [ 16 ] and GraphSage [ 11 ] , improved the state-of-the-art in node classification with their elegant yet powerful designs . However , as GNNs leverage the homophily of nodes in both node features and link structures to conduct the inference , they are vulnerable to the perturbation on graphs [ 4 , 40 , 41 ] . Robust GNNs , aiming at reducing the degeneration in GNNs caused by graph perturbation , are gaining attention these days . Current robust GNNs focus on the sensitivity towards modifications on node features [ 3 , 42 , 15 ] or adding/removing edges on the graph [ 37 ] . However , neither of these two types recapitulates the missing neighbor problem , which affects both the feature distribution and structure distribution . To obtain a node classifier with good generalizability , the development of domain adaptive GNN sheds light on adapting a GNN model trained on the source domain to the target domain by leveraging underlying structural consistency [ 38 , 36 , 28 ] . However , in the distributed system we consider , data owners have subgraphs with heterogeneous feature and structure distributions . Moreover , direct information exchanges among subgraphs , such as message passing , are fully blocked due to the missing cross-subgraph links . The violation of the domain adaptive GNNs ’ assumptions on alignable nodes and cross-domain structural consistency denies their usage in the distributed subgraph system . Federated learning . FL is proposed for cross-institutional collaborative learning without sharing raw data [ 17 , 35 , 21 ] . FedAvg [ 21 ] is an efficient and well-studied FL method . Similar to most FL methods , it is originally proposed for traditional machine learning problems [ 35 ] to allow collaborative training on silo data through local updating and global aggregation . The ecently proposed meta-learning framework [ 9 , 23 , 14 ] that exploits information from different data sources to obtain a general model attracts FL researchers [ 8 ] . However , meta-learning aims to learn general models that easily adapt to different local tasks , while we learn a generalizable model from diverse data owners to assist in solving a global task . In the distributed subgraph system , to obtain a globally applicable model without sharing local graph data , we borrow the idea of FL to collaboratively train GNNs . Federated graph learning . Recent researchers have made some progress in federated graph learning . There are existing FL frameworks designed for the graph data learning task [ 12 , 27 , 30 ] . [ 12 ] design graph-level FL schemes with graph datasets dispersed over multiple data owners , which are inapplicable to our distributed subgraph system construction . [ 27 ] proposes an FL method for the recommendation problem with each data owner learning on a subgraph of the whole recommendation user-item graph . It considers a different scenario assuming subgraphs have overlapped items ( nodes ) , and the user-item interactions ( edges ) are distributed but completely stored in the system , which ignores the possible cross-subgraph information lost in real-world scenarios . However , we study a more challenging yet realistic case in the distributed subgraph system , where cross-subgraph edges are totally missing . In this work , we consider the commonly existing yet not studied scenario , i.e. , distributed subgraph system with missing cross-subgraph edges . Under this scenario , we focus on obtaining a globally applicable node classifier through FL on distributed subgraphs . 3 FedSage . In this section , we first illustrate the definition of the distributed subgraph system derived from real-world application scenarios . Based on this system , we then formulate our novel subgraph FL framework and a vanilla solution called FedSage . 3.1 Subgraphs Distributed in Local Systems . Notation . We denote a global graph asG = { V , E , X } , where V is the node set , X is the respective node feature set , and E is the edge set . In the FL system , we have the central server S , and M data owners with distributed subgraphs . Gi = { Vi , Ei , Xi } is the subgraph owned by Di , for i ∈ [ M ] . Problem setup . For the whole system , we assume V = V1∪· · ·∪VM . To simulate the scenario with most missing links , we assume no overlapping nodes shared across data owners , namely Vi ∩ Vj = ∅ for ∀i , j ∈ [ M ] and i 6= j . Note that the central server S only maintains a graph mining model with no actual graph data stored . Any data owner Di can not directly retrieve u ∈ Vj from another data owner Dj . Therefore , for an edge ev , u ∈ E , where v ∈ Vi and u ∈ Vj , ev , u /∈ Ei ∪Ej , that is , ev , u might exist in reality but is not stored anywhere in the whole system . For the global graph G = { V , E , X } , every node v ∈ V has its features xv ∈ X and one label yv ∈ Y for the downstream task , e.g. , node classification . Note that for v ∈ V , v ’ s feature xv ∈ Rdx and respective label yv is a dy-dimensional one-hot vector . In a typical GNN , predicting a node ’ s label requires an ego-graph of the queried node . For a node v from graph G , we denote the queried ego-graph of v as G ( v ) , and ( G ( v ) , yv ) ∼ DG . With subgraphs distributed in the system defined above , we formulate our goal as follows . Goal . The system exploits an FL framework to collaboratively learn on isolated subgraphs in all data owners , without raw graph data sharing , to obtain a global node classifier F . The learnable weights φ in F is optimized for queried ego-graphs following the distribution of ones drawn from the global graph G. We formalize the problem as finding φ∗ that minimizes the aggregated risk φ∗ = arg minR ( F ( φ ) ) = 1 M M∑ i Ri ( Fi ( φ ) ) ) , whereRi is the local empirical risk defined as Ri ( Fi ( φ ) ) : = E ( Gi , Yi ) ∼DGi [ ` ( Fi ( φ ; Gi ) , Yi ) ) ] , where ` is a task-specific loss function ` : = 1 |Vi| ∑ v∈Vi l ( φ ; Gi ( v ) , yv ) .

Summary: This paper proposes a federated learning framework for training federated GNNs under the scenario where each dataholder holds disjoint a disjoint subgraph. The paper introduces FedSAGE, which extends GraphSAGE to the federated setting, and FedSAGE+, which further enhances FedSAGE with a missing neighbor generator. The key insight of FedSAGE+ is that the links between subgraphs will be missing, and the neighbor generator is trained to generate missing neighbors. The authors demonstrate their framework using experiments on Cora, Citeseer, Pubmed and MSAcademic, where FedSAGE+ shows improvement over LocalSAGE without the missing neighbor generation.