Transfer Learning of Graph Neural Networks with Ego-graph Information Maximization

1 Introduction . Graph neural networks ( GNNs ) have been intensively studied recently [ 29 , 26 , 39 , 68 ] , due to their established performance towards various real-world tasks [ 15 , 69 , 53 ] , as well as close connections to spectral graph theory [ 12 , 9 , 16 ] . While most GNN architectures are not very complicated , the training of GNNs can still be costly regarding both memory and computation resources on real-world large-scale graphs [ 10 , 63 ] . Moreover , it is intriguing to transfer learned structural information across different graphs and even domains in settings like few-shot learning [ 56 , 44 , 25 ] . Therefore , several very recent studies have been conducted on the transferability of GNNs [ 21 , 23 , 22 , 59 , 31 , 3 , 47 ] . However , it is unclear in what situations the models will excel or fail especially when the pre-training and fine-tuning tasks are different . To provide rigorous analysis and guarantee on the transferability of GNNs , we focus on the setting of direct-transfering between the source and target graphs , under an analogous setting of “ domain adaptation ” [ 7 , 59 ] . In this work , we establish a theoretically grounded framework for the transfer learning of GNNs , and leverage it to design a practically transferable GNN model . Figure 1 gives an overview of our framework . It is based on a novel view of a graph as samples from the joint distribution of its k-hop ego-graph structures and node features , which allows us to define graph information and similarity , ∗These two authors contribute equally . 1Code and processed data are available at https : //github.com/GentleZhu/EGI . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) , Online . so as to analyze GNN transferability ( §3 ) . This view motivates us to design EGI , a novel GNN training objective based on ego-graph information maximization , which is effective in capturing the graph information as we define ( §3.1 ) . Then we further specify the requirement on transferable node features and analyze the transferability of EGI that is dependent on the local graph Laplacians of source and target graphs ( §3.2 ) . All of our theoretical conclusions have been directly validated through controlled synthetic experiments ( Table 1 ) , where we use structural-equivalent role identification in an direct-transfering setting to analyze the impacts of different model designs , node features and source-target structure similarities on GNN transferability . In §4 , we conduct real-world experiments on multiple publicly available network datasets . On the Airport and Gene graphs ( §4.1 ) , we closely follow the settings of our synthetic experiments and observe consistent but more detailed results supporting the design of EGI and the utility of our theoretical analysis . On the YAGO graphs ( §4.2 ) , we further evaluate EGI on the more generalized and practical setting of transfer learning with task-specific fine-tuning . We find our theoretical insights still indicative in such scenarios , where EGI consistently outperforms state-of-the-art GNN representation and transfer learning frameworks with significant margins . 2 Related Work . Representation learning on graphs has been studied for decades , with earlier spectral-based methods [ 6 , 46 , 52 ] theoretically grounded but hardly scaling up to graphs with over a thousand of nodes . With the emergence of neural networks , unsupervised network embedding methods based on the Skip-gram objective [ 37 ] have replenished the field [ 51 , 14 , 42 , 45 , 66 , 62 , 65 ] . Equipped with efficient structural sampling ( random walk , neighborhood , etc . ) and negative sampling schemes , these methods are easily parallelizable and scalable to graphs with thousands to millions of nodes . However , these models are essentially transductive as they compute fully parameterized embeddings only for nodes seen during training , which are impossible to be transfered to unseen graphs . More recently , researchers introduce the family of graph neural networks ( GNNs ) that are capable of inductive learning and generalizing to unseen nodes given meaningful node features [ 29 , 12 , 15 , 67 ] . Yet , most existing GNNs require task-specific labels for training in a semi-supervised fashion to achieve satisfactory performance [ 29 , 15 , 53 , 64 ] , and their usage is limited to single graphs where the downstream task is fixed . To this end , several unsupervised GNNs are presented , such as the auto-encoder-based ones like VGAE [ 28 ] and GNFs [ 35 ] , as well as the deep-infomax-based ones like DGI [ 54 ] and InfoGraph [ 50 ] . Their potential in the transfer learning of GNN remains unclear when the node features and link structures vary across different graphs . Although the architectures of popular GNNs such as GCN [ 29 ] may not be very complicated compared with heavy vision and language models , training a dedicated GNN for each graph can still be cumbersome [ 10 , 63 ] . Moreover , as pre-training neural networks are proven to be successful in other domains [ 13 , 18 ] , the idea is intriguing to transfer well-trained GNNs from relevant source graphs to improve the modeling of target graphs or enable few-shot learning [ 59 , 31 , 3 ] when labeled data are scarce . In light of this , pioneering works have studied both generative [ 22 ] and discriminative [ 21 , 23 ] GNN pre-training schemes . Though Graph Contrastive Coding [ 43 ] shares the most similar view towards graph structures as us , it utilizes contrastive learning across all graphs instead of focusing on the transfer learning between any specific pairs . On the other hand , unsupervised domain adaptive GCNs [ 59 ] study the domain adaption problem only when the source and target tasks are homogeneous . Most previous pre-training and self-supervised GNNs lack a rigorous analysis towards their transferability and thus have unpredictable effectiveness . The only existing theoretical work on GNN transferability studies the performance of GNNs across different permutations of a single original graph [ 33 , 34 ] and the tradeoff between discriminability and transferability of GNNs [ 47 ] . We , instead , are the first to rigorously study the more practical setting of transferring GNNs across pairs of different source and target graphs . 3 Transferable Graph Neural Networks . In this paper , we design a more transferable training objective for GNN ( EGI ) based on our novel view of essential graph information ( §3.1 ) . We then analyze its transferability as the gap between its abilities to model the source and target graphs , based on their local graph Laplacians ( §3.2 ) . Based on the connection between GNN and spectral graph theory [ 29 ] , we describe the output of a GNN as a combination of its input node features X , fixed graph Laplacian L and learnable graph filters Ψ . The goal of training a GNN is then to improve its utility by learning the graph filters that are compatible with the other two components towards specific tasks . In the graph transfer learning setting where downstream tasks are often unknown during pre-training , we argue that the general utility of a GNN should be optimized and quantified w.r.t . its ability of capturing the essential graph information in terms of the joint distribution of its topology structures and node features , which motivates us to design a novel ego-graph information maximization model ( EGI ) ( §3.1 ) . The general transferability of a GNN is then quantified by the gap between its abilities to model the source and target graphs . Under reasonable requirements such as using structurerespecting node features as the GNN input , we analyze this gap for EGI based on the structural difference between two graphs w.r.t . their local graph Laplacians ( §3.2 ) . 3.1 Transferable GNN via Ego-graph Information Maximization . In this work , we focus on the direct-transfering setting where a GNN is pre-trained on a source graph Ga in an unsupervised fashion and applied on a target graph Gb without fine-tuning.2 Consider a graph G = { V , E } , where the set of nodes V are associated with certain features X and the set of edges E form graph structures . Intuitively , the transfer learning will be successful only if both the features and structures of Ga and Gb are similar in some ways , so that the graph filters of a GNN learned on Ga are compatible with the features and structures of Gb . Graph kernels [ 57 , 8 , 30 , 38 ] are well-known for their capability of measuring similarity between pair of graphs . Motivated by k-hop subgraph kernels [ 4 ] , we introduce a novel view of a graph as samples from the joint distribution of its k-hop ego-graph structures and node features . Since GNN essentially encodes such k-hop ego graph samples , this view allows us to give concrete definitions towards structural information of graphs in the transfer learning setting , which facilitates the measuring of similarity ( difference ) among graphs . Yet , none of the existing GNN training objectives are capable of recovering such distributional signals of ego graphs . To this end , we design Ego-Graph Information maximization ( EGI ) , which alternatively reconstructs the k-hop ego-graph of each center node via mutual information maximization [ 20 ] . Definition 3.1 ( K-hop ego-graph ) . We call a graph gi = { V ( gi ) , E ( gi ) } a k-hop ego-graph centered at node vi if it has a k-layer centroid expansion [ 4 ] such that the greatest distance between vi and 2In the experiments , we show our model to be generalizable to the more practical settings with task-specific pre-training and fine-tuning , while the study of rigorous bound in such scenarios is left as future work . any other nodes in the ego-graph is k , i.e . ∀vj ∈ V ( gi ) , |d ( vi , vj ) | ≤ k , where d ( vi , vj ) is the graph distance between vi and vj . In this paper , we use directed k-hop ego-graph and its direction is decided by whether it is composed of incoming or outgoing edges to the center node , i.e. , gi and g̃i . The results apply trivially to undirected graphs with gi = g̃i . Definition 3.2 ( Structural information ) . Let G be a topological space of sub-graphs , we view a graph G as samples of k-hop ego-graphs { gi } ni=1 drawn i.i.d . from G with probability µ , i.e. , gi i.i.d.∼ µ ∀i = 1 , · · · , n. The structural information of G is then defined to be the set of k-hop ego-graphs of { gi } ni=1 and their empirical distribution . As shown in Figure 1 , three graphs G0 , G1 and G2 are characterized by a set of 1-hop ego-graphs and their empirical distributions , which allows us to quantify the structural similarity among graphs as shown in §3.2 ( i.e. , G0 is more similar to G1 than G2 under such characterization ) . In practice , the nodes in a graph G are characterized not only by their k-hop ego-graph structures but also their associated node features . Therefore , G should be regarded as samples { ( gi , xi ) } drawn from the joint distribution P on the product space of G and a node feature space X . Ego-Graph Information Maximization . Given a set of ego-graphs { ( gi , xi ) } i drawn from an empirical joint distribution ( gi , xi ) ∼ P. We aim to train an GNN encoder Ψ to maximize the mutual informaion ( MI ( gi , Ψ ( gi , xi ) ) ) between the defined structural information gi3 ( i.e . k-hop ego-graph ) and node embedding zi = Ψ ( gi , xi ) . To maximize the MI , another discriminator D ( gi , zi ) : E ( gi ) × zi → R+ is introduced to compute the probability of an edge e belongs to the given ego-graph gi . We use the Jensen-Shannon MI estimator [ 20 ] in the EGI objective , LEGI = −MI ( JSD ) ( G , Ψ ) = 1N N∑ i=1 [ sp ( D ( gi , z′i ) ) + sp ( −D ( gi , zi ) ) ] , ( 1 ) where sp ( x ) = log ( 1+ex ) is the softplus function and ( gi , z′i ) is randomly drawn from the product of marginal distributions , i.e . z′i = Ψ ( gi′ , xi′ ) , ( gi′ , xi′ ) ∼ P , i′ 6= i . In general , we can also randomly draw negative g′i in the topological space , while enumerating all possible graphs gi′ leads to high computation cost . In Eq . 1 , the computation of D on E ( gi ) depends on the node orders . Following the common practice in graph generation [ 70 ] , we characterize the decision process of D with a fixed graph ordering , i.e. , the BFS-ordering π over edges E ( gi ) . D = f ◦ Φ is composed by another GNN encoder Φ and scoring function f over an edge sequence Eπ : { e1 , e2 , ... , en } , which makes predictions on the BFS-ordered edges . 3Later in section 3.2 , we will discuss the equivalence between MI ( gi , zi ) and MI ( ( gi , xi ) , zi ) when node feature is structure-respecting . Recall our previous definition on the direction of k-hop ego-graph , the center node encoder Ψ receives pairs of ( gi , xi ) while the neighbor node encoder Φ in discriminator D receives ( g̃i , xi ) . Both encoders are parameterized as GNNs , Ψ ( gi , xi ) = GNNΨ ( Ai , Xi ) , Φ ( g̃i , xi ) = GNNΦ ( A′i , Xi ) , where Ai , A′i is the adjacency matrix with self-loops of gi and g̃i , respectively . The self-loops are added following the common design of GNNs , which allows the convolutional node embeddings to always incorporate the influence of the center node . Ai = A′i ᵀ . The output of Ψ , i.e. , zi ∈ Rn , is the center node embedding , while Φ outputs representation H ∈ R|gi|×n for neighbor nodes in the ego-graph . Once node representation H is computed , we now describe the scoring function f . For each of the node pair ( p , q ) ∈ Eπ , hp is the source node representation from Φ , xq is the destination node features . The scoring function is , f ( hp , xq , zi ) = σ ( UT · τ ( WT [ hp||xq||zi ] ) ) , ( 2 ) where σ and τ are Sigmoid and ReLU activation functions . Thus , the discriminator D is asked to distinguish a positive ( ( p , q ) , zi ) and negative pair ( ( p , q ) , z′i ) ) for each edge in gi . D ( gi , zi ) = ∑ ( p , q ) ∈Eπ log f ( hp , xq , zi ) , D ( gi , z′i ) = Eπ∑ ( p , q ) log f ( hp , xq , z ′ i ) . ( 3 ) There are two types of edges ( p , q ) in our consideration of node orders , type-a - the edges across different hops ( from the center node ) , and type-b - the edges within the same hop ( from the center node ) . The aforementioned BFS-based node ordering guarantees that Eq . 3 is sensitive to the ordering of type-a edges , and invariant to the ordering of type-b edges , which is consistent with the requirement of our theoretical analysis on ∆D . Due to the fact that the output of a k-layer GNN only depends on a k-hop ego-graph for both encoders Ψ and Φ , EGI can be trained in parallel by sampling batches of gi ’ s . Besides , the training objective of EGI is transferable as long as ( gi , xi ) across source graph Ga and Gb satisfies the conditions given in §3.2 . More model details in Appendix §B and source code in the Supplementary Materials . Connection with existing work . To provide more insights into the EGI objective , we also present it as a dual problem of ego-graph reconstruction . Recall our definition of ego-graph mutual information MI ( gi , Ψ ( gi , xi ) ) . It can be related to an ego-graph reconstruction loss R ( gi|Ψ ( gi , xi ) ) as max MI ( gi , Ψ ( gi , xi ) ) = H ( gi ) −H ( gi|Ψ ( gi , xi ) ) ≤ H ( gi ) −R ( gi|Ψ ( gi , xi ) ) . ( 4 ) When EGI is maximizing the mutual information , it simultaneously minimizes the upper error bound of reconstructing an ego-graph gi . In this view , the key difference between EGI and VGAE [ 28 ] is they assume each edge in a graph to be observed independently during the reconstruction . While in EGI , edges in an ego-graph are observed jointly during the GNN decoding . Moreover , existing mutual information based GNNs such as DGI [ 54 ] and GMI [ 41 ] explicitly measure the mutual information between node features x and GNN output Ψ . In this way , they tend to capture node features instead of graph structures , which we deem more essential in graph transfer learning as discussed in §3.2 . Use cases of EGI framework . In this paper , we focus on the classical domain adaption ( directtransferring ) setting [ 7 ] , where no target domain labels are available and transferability is measured by the performance discrepancy without fine-tuning . In this setting , the transferability of EGI is theoretically guaranteed by Theorem 3.1 . In §4.1 , we validated this with the airport datasets . Beyond direct-transferring , EGI is also useful in the more generalized and practical setting of transfer learning with fine-tuning , which we introduced in §4.2 and validated with the YAGO datasets . In this setting , the transferability of EGI is not rigorously studied yet , but is empirically shown promising . Supportive observations . In the first three columns of our synthetic experimental results ( Table 1 ) , in both cases of transfering GNNs between similar graphs ( F-F ) and dissimilar graphs ( B-F ) , EGI significantly outperforms all competitors when using node degree one-hot encoding as transferable node features . In particular , the performance gains over the untrained GIN show the effectiveness of training and transfering , and our gains are always larger than the two state-of-the-art unsupervised GNNs . Such results clearly indicate advantageous structure preserving capability and transferability of EGI .

Summary: This paper develops a novel measure for assessing the transferability of graph neural network models to new data sets. The measure is based on a decomposition of graphs into 'ego networks' (essentially, a distribution of $k$-hop subgraph, extracted from a given larger graph). Transferability is then assessed by means of a spectral criterion using the graph Laplacian. Experiments demonstrate the utility in assessing transferability in such a manner, as the new measure appears to be aligned with improvements in predictive performance.