Subgraph Federated Learning with Missing Neighbor Generation

1 Introduction . Graph mining leverages links among connected nodes in graphs to conduct inference . Recently , graph neural networks ( GNNs ) have gained applause with impressing performance and generalizability in many graph mining tasks [ 29 , 11 , 16 , 20 , 32 ] . Similar to machine learning tasks in other domains , attaining a well-performed GNN model requires its training data to not only be sufficient but also follow the similar distribution as general queries . While in reality , data owners often collect limited and biased graphs and can not observe the global distribution . With heterogeneous subgraphs separately stored in local data owners , accomplishing a globally applicable GNN requires collaboration . Federated learning ( FL ) [ 17 , 35 ] , targeting at training machine learning models with data distributed in multiple local systems to resolve the information-silo problem , has shown its advantage in enhancing the performance and generalizability of the collaboratively trained models without the need of sharing any actual data . For example , FL has been devised in computer vision ( CV ) and natural language processing ( NLP ) to allow the joint training of powerful and generalizable deep convolutional neural networks and language models on separately stored datasets of images and texts [ 19 , 6 , 18 , 39 , 13 ] . Motivating Scenario . Taking the healthcare system as an example , as shown in Fig . 1 , residents of a city may go to different hospitals for various reasons . As a result , their healthcare data , such as demographics and living conditions , as well as patient interactions , such as co-staying in a sickroom and co-diagnosis of a disease , are stored only within the hospitals they visit . When any healthcare problem is to be studied in the whole city , e.g. , the prediction of infections when a pandemic occurs , ∗Corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . a single powerful graph mining model is needed to conduct effective inference over the entire global patient network , which contains all subgraphs from different hospitals . However , it is rather difficult to let all hospitals share their patient networks with others to train the graph mining model due to conflicts of interests and privacy concerns . In such scenarios , it is desirable to train a powerful and generalizable graph mining model over multiple distributed subgraphs without actual data sharing . However , this novel yet realistic setting brings two unique technical challenges , which have never been explored so far . Challenge 1 : How to jointly learn from multiple local subgraphs ? In our considered scenario , the global graph is distributed into a set of small subgraphs with heterogeneous feature and structure distributions . Training a separate graph mining model on each subgraph may not capture the global data distribution and is also prone to overfitting . Moreover , it is unclear how to integrate multiple graph mining models into a universally applicable one that can handle any queries from the underlying global graph . Solution 1 : FedSage : Training GraphSage with FedAvg . To attain a powerful and generalizable graph mining model from small and biased subgraphs distributed in multiple local owners , we develop a framework of subgraph federated learning , specifically , with the vanilla mechanism of FedAvg [ 21 ] . As for the graph mining model , we resort to GraphSage [ 11 ] , due to its advantages of inductiveness and scalability . We term this framework as FedSage . Challenge 2 : How to deal with missing links across local subgraphs ? Unlike distributed systems in other domains such as CV and NLP , whose data samples of images and texts are isolated and independent , data samples in graphs are connected and correlated . Most importantly , in a subgraph federated learning system , data samples in each subgraph can potentially have connections to those in other subgraphs . These connections carrying important information of node neighborhoods and serving as bridges among the data owners , however , are never directly captured by any data owner . Solution 2 : FedSage+ : Generating missing neighbors along FedSage . To deal with crosssubgraph missing links , we add a missing neighbor generator on top of FedSage and propose a novel FedSage+ model . Specifically , for each data owner , instead of training the GraphSage model on the original subgraph , it first mends the subgraph with generated cross-subgraph missing neighbors and then applies FedSage on the mended subgraph . To obtain the missing neighbor generator , each data owner impairs the subgraph by randomly holding out some nodes and related links and then trains the generator based on the held-out neighbors . Training the generator on an individual local subgraph enables it to generate potential missing links within the subgraph . Further training the generator in our subgraph FL setting allows it to generate missing neighbors across distributed subgraphs . We conduct experiments on four real-world datasets with different numbers of data owners to better simulate the application scenarios . According to our results , both of our models outperform locally trained classifiers in all scenarios . Compared to FedSage , FedSage+ further promotes the performance of the outcome classifier . Further in-depth model analysis shows the convergence and generalization ability of our frameworks , which is corroborated by our theoretical analysis in the end . 2 Related works . Graph mining . Graph mining emerges its significance in analyzing the informative graph data , which range from social networks to gene interaction networks [ 31 , 33 , 34 , 24 ] . One of the most frequently applied tasks on graph data is node classification . Recently , graph neural networks ( GNNs ) , e.g. , graph convolutional networks ( GCN ) [ 16 ] and GraphSage [ 11 ] , improved the state-of-the-art in node classification with their elegant yet powerful designs . However , as GNNs leverage the homophily of nodes in both node features and link structures to conduct the inference , they are vulnerable to the perturbation on graphs [ 4 , 40 , 41 ] . Robust GNNs , aiming at reducing the degeneration in GNNs caused by graph perturbation , are gaining attention these days . Current robust GNNs focus on the sensitivity towards modifications on node features [ 3 , 42 , 15 ] or adding/removing edges on the graph [ 37 ] . However , neither of these two types recapitulates the missing neighbor problem , which affects both the feature distribution and structure distribution . To obtain a node classifier with good generalizability , the development of domain adaptive GNN sheds light on adapting a GNN model trained on the source domain to the target domain by leveraging underlying structural consistency [ 38 , 36 , 28 ] . However , in the distributed system we consider , data owners have subgraphs with heterogeneous feature and structure distributions . Moreover , direct information exchanges among subgraphs , such as message passing , are fully blocked due to the missing cross-subgraph links . The violation of the domain adaptive GNNs ’ assumptions on alignable nodes and cross-domain structural consistency denies their usage in the distributed subgraph system . Federated learning . FL is proposed for cross-institutional collaborative learning without sharing raw data [ 17 , 35 , 21 ] . FedAvg [ 21 ] is an efficient and well-studied FL method . Similar to most FL methods , it is originally proposed for traditional machine learning problems [ 35 ] to allow collaborative training on silo data through local updating and global aggregation . The ecently proposed meta-learning framework [ 9 , 23 , 14 ] that exploits information from different data sources to obtain a general model attracts FL researchers [ 8 ] . However , meta-learning aims to learn general models that easily adapt to different local tasks , while we learn a generalizable model from diverse data owners to assist in solving a global task . In the distributed subgraph system , to obtain a globally applicable model without sharing local graph data , we borrow the idea of FL to collaboratively train GNNs . Federated graph learning . Recent researchers have made some progress in federated graph learning . There are existing FL frameworks designed for the graph data learning task [ 12 , 27 , 30 ] . [ 12 ] design graph-level FL schemes with graph datasets dispersed over multiple data owners , which are inapplicable to our distributed subgraph system construction . [ 27 ] proposes an FL method for the recommendation problem with each data owner learning on a subgraph of the whole recommendation user-item graph . It considers a different scenario assuming subgraphs have overlapped items ( nodes ) , and the user-item interactions ( edges ) are distributed but completely stored in the system , which ignores the possible cross-subgraph information lost in real-world scenarios . However , we study a more challenging yet realistic case in the distributed subgraph system , where cross-subgraph edges are totally missing . In this work , we consider the commonly existing yet not studied scenario , i.e. , distributed subgraph system with missing cross-subgraph edges . Under this scenario , we focus on obtaining a globally applicable node classifier through FL on distributed subgraphs . 3 FedSage . In this section , we first illustrate the definition of the distributed subgraph system derived from real-world application scenarios . Based on this system , we then formulate our novel subgraph FL framework and a vanilla solution called FedSage . 3.1 Subgraphs Distributed in Local Systems . Notation . We denote a global graph asG = { V , E , X } , where V is the node set , X is the respective node feature set , and E is the edge set . In the FL system , we have the central server S , and M data owners with distributed subgraphs . Gi = { Vi , Ei , Xi } is the subgraph owned by Di , for i ∈ [ M ] . Problem setup . For the whole system , we assume V = V1∪· · ·∪VM . To simulate the scenario with most missing links , we assume no overlapping nodes shared across data owners , namely Vi ∩ Vj = ∅ for ∀i , j ∈ [ M ] and i 6= j . Note that the central server S only maintains a graph mining model with no actual graph data stored . Any data owner Di can not directly retrieve u ∈ Vj from another data owner Dj . Therefore , for an edge ev , u ∈ E , where v ∈ Vi and u ∈ Vj , ev , u /∈ Ei ∪Ej , that is , ev , u might exist in reality but is not stored anywhere in the whole system . For the global graph G = { V , E , X } , every node v ∈ V has its features xv ∈ X and one label yv ∈ Y for the downstream task , e.g. , node classification . Note that for v ∈ V , v ’ s feature xv ∈ Rdx and respective label yv is a dy-dimensional one-hot vector . In a typical GNN , predicting a node ’ s label requires an ego-graph of the queried node . For a node v from graph G , we denote the queried ego-graph of v as G ( v ) , and ( G ( v ) , yv ) ∼ DG . With subgraphs distributed in the system defined above , we formulate our goal as follows . Goal . The system exploits an FL framework to collaboratively learn on isolated subgraphs in all data owners , without raw graph data sharing , to obtain a global node classifier F . The learnable weights φ in F is optimized for queried ego-graphs following the distribution of ones drawn from the global graph G. We formalize the problem as finding φ∗ that minimizes the aggregated risk φ∗ = arg minR ( F ( φ ) ) = 1 M M∑ i Ri ( Fi ( φ ) ) ) , whereRi is the local empirical risk defined as Ri ( Fi ( φ ) ) : = E ( Gi , Yi ) ∼DGi [ ` ( Fi ( φ ; Gi ) , Yi ) ) ] , where ` is a task-specific loss function ` : = 1 |Vi| ∑ v∈Vi l ( φ ; Gi ( v ) , yv ) .

Summary: This paper proposed a generalized node classification model over a distributed sub-graph system without data-sharing. The FedSage is a globally shared K-layer graph classifier model, trained with average gradients over distributed sub-graphs without data-sharing. The awareness of potentially existing links between sub-graphs motivate the authors to design a  missing neighbor model to help improve the performance of the node classifier model named FedSage+, which has been empirically shown to outperform the FedSage.