Transfer Learning of Graph Neural Networks with Ego-graph Information Maximization

1 Introduction . Graph neural networks ( GNNs ) have been intensively studied recently [ 29 , 26 , 39 , 68 ] , due to their established performance towards various real-world tasks [ 15 , 69 , 53 ] , as well as close connections to spectral graph theory [ 12 , 9 , 16 ] . While most GNN architectures are not very complicated , the training of GNNs can still be costly regarding both memory and computation resources on real-world large-scale graphs [ 10 , 63 ] . Moreover , it is intriguing to transfer learned structural information across different graphs and even domains in settings like few-shot learning [ 56 , 44 , 25 ] . Therefore , several very recent studies have been conducted on the transferability of GNNs [ 21 , 23 , 22 , 59 , 31 , 3 , 47 ] . However , it is unclear in what situations the models will excel or fail especially when the pre-training and fine-tuning tasks are different . To provide rigorous analysis and guarantee on the transferability of GNNs , we focus on the setting of direct-transfering between the source and target graphs , under an analogous setting of “ domain adaptation ” [ 7 , 59 ] . In this work , we establish a theoretically grounded framework for the transfer learning of GNNs , and leverage it to design a practically transferable GNN model . Figure 1 gives an overview of our framework . It is based on a novel view of a graph as samples from the joint distribution of its k-hop ego-graph structures and node features , which allows us to define graph information and similarity , ∗These two authors contribute equally . 1Code and processed data are available at https : //github.com/GentleZhu/EGI . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) , Online . so as to analyze GNN transferability ( §3 ) . This view motivates us to design EGI , a novel GNN training objective based on ego-graph information maximization , which is effective in capturing the graph information as we define ( §3.1 ) . Then we further specify the requirement on transferable node features and analyze the transferability of EGI that is dependent on the local graph Laplacians of source and target graphs ( §3.2 ) . All of our theoretical conclusions have been directly validated through controlled synthetic experiments ( Table 1 ) , where we use structural-equivalent role identification in an direct-transfering setting to analyze the impacts of different model designs , node features and source-target structure similarities on GNN transferability . In §4 , we conduct real-world experiments on multiple publicly available network datasets . On the Airport and Gene graphs ( §4.1 ) , we closely follow the settings of our synthetic experiments and observe consistent but more detailed results supporting the design of EGI and the utility of our theoretical analysis . On the YAGO graphs ( §4.2 ) , we further evaluate EGI on the more generalized and practical setting of transfer learning with task-specific fine-tuning . We find our theoretical insights still indicative in such scenarios , where EGI consistently outperforms state-of-the-art GNN representation and transfer learning frameworks with significant margins . 2 Related Work . Representation learning on graphs has been studied for decades , with earlier spectral-based methods [ 6 , 46 , 52 ] theoretically grounded but hardly scaling up to graphs with over a thousand of nodes . With the emergence of neural networks , unsupervised network embedding methods based on the Skip-gram objective [ 37 ] have replenished the field [ 51 , 14 , 42 , 45 , 66 , 62 , 65 ] . Equipped with efficient structural sampling ( random walk , neighborhood , etc . ) and negative sampling schemes , these methods are easily parallelizable and scalable to graphs with thousands to millions of nodes . However , these models are essentially transductive as they compute fully parameterized embeddings only for nodes seen during training , which are impossible to be transfered to unseen graphs . More recently , researchers introduce the family of graph neural networks ( GNNs ) that are capable of inductive learning and generalizing to unseen nodes given meaningful node features [ 29 , 12 , 15 , 67 ] . Yet , most existing GNNs require task-specific labels for training in a semi-supervised fashion to achieve satisfactory performance [ 29 , 15 , 53 , 64 ] , and their usage is limited to single graphs where the downstream task is fixed . To this end , several unsupervised GNNs are presented , such as the auto-encoder-based ones like VGAE [ 28 ] and GNFs [ 35 ] , as well as the deep-infomax-based ones like DGI [ 54 ] and InfoGraph [ 50 ] . Their potential in the transfer learning of GNN remains unclear when the node features and link structures vary across different graphs . Although the architectures of popular GNNs such as GCN [ 29 ] may not be very complicated compared with heavy vision and language models , training a dedicated GNN for each graph can still be cumbersome [ 10 , 63 ] . Moreover , as pre-training neural networks are proven to be successful in other domains [ 13 , 18 ] , the idea is intriguing to transfer well-trained GNNs from relevant source graphs to improve the modeling of target graphs or enable few-shot learning [ 59 , 31 , 3 ] when labeled data are scarce . In light of this , pioneering works have studied both generative [ 22 ] and discriminative [ 21 , 23 ] GNN pre-training schemes . Though Graph Contrastive Coding [ 43 ] shares the most similar view towards graph structures as us , it utilizes contrastive learning across all graphs instead of focusing on the transfer learning between any specific pairs . On the other hand , unsupervised domain adaptive GCNs [ 59 ] study the domain adaption problem only when the source and target tasks are homogeneous . Most previous pre-training and self-supervised GNNs lack a rigorous analysis towards their transferability and thus have unpredictable effectiveness . The only existing theoretical work on GNN transferability studies the performance of GNNs across different permutations of a single original graph [ 33 , 34 ] and the tradeoff between discriminability and transferability of GNNs [ 47 ] . We , instead , are the first to rigorously study the more practical setting of transferring GNNs across pairs of different source and target graphs . 3 Transferable Graph Neural Networks . In this paper , we design a more transferable training objective for GNN ( EGI ) based on our novel view of essential graph information ( §3.1 ) . We then analyze its transferability as the gap between its abilities to model the source and target graphs , based on their local graph Laplacians ( §3.2 ) . Based on the connection between GNN and spectral graph theory [ 29 ] , we describe the output of a GNN as a combination of its input node features X , fixed graph Laplacian L and learnable graph filters Ψ . The goal of training a GNN is then to improve its utility by learning the graph filters that are compatible with the other two components towards specific tasks . In the graph transfer learning setting where downstream tasks are often unknown during pre-training , we argue that the general utility of a GNN should be optimized and quantified w.r.t . its ability of capturing the essential graph information in terms of the joint distribution of its topology structures and node features , which motivates us to design a novel ego-graph information maximization model ( EGI ) ( §3.1 ) . The general transferability of a GNN is then quantified by the gap between its abilities to model the source and target graphs . Under reasonable requirements such as using structurerespecting node features as the GNN input , we analyze this gap for EGI based on the structural difference between two graphs w.r.t . their local graph Laplacians ( §3.2 ) . 3.1 Transferable GNN via Ego-graph Information Maximization . In this work , we focus on the direct-transfering setting where a GNN is pre-trained on a source graph Ga in an unsupervised fashion and applied on a target graph Gb without fine-tuning.2 Consider a graph G = { V , E } , where the set of nodes V are associated with certain features X and the set of edges E form graph structures . Intuitively , the transfer learning will be successful only if both the features and structures of Ga and Gb are similar in some ways , so that the graph filters of a GNN learned on Ga are compatible with the features and structures of Gb . Graph kernels [ 57 , 8 , 30 , 38 ] are well-known for their capability of measuring similarity between pair of graphs . Motivated by k-hop subgraph kernels [ 4 ] , we introduce a novel view of a graph as samples from the joint distribution of its k-hop ego-graph structures and node features . Since GNN essentially encodes such k-hop ego graph samples , this view allows us to give concrete definitions towards structural information of graphs in the transfer learning setting , which facilitates the measuring of similarity ( difference ) among graphs . Yet , none of the existing GNN training objectives are capable of recovering such distributional signals of ego graphs . To this end , we design Ego-Graph Information maximization ( EGI ) , which alternatively reconstructs the k-hop ego-graph of each center node via mutual information maximization [ 20 ] . Definition 3.1 ( K-hop ego-graph ) . We call a graph gi = { V ( gi ) , E ( gi ) } a k-hop ego-graph centered at node vi if it has a k-layer centroid expansion [ 4 ] such that the greatest distance between vi and 2In the experiments , we show our model to be generalizable to the more practical settings with task-specific pre-training and fine-tuning , while the study of rigorous bound in such scenarios is left as future work . any other nodes in the ego-graph is k , i.e . ∀vj ∈ V ( gi ) , |d ( vi , vj ) | ≤ k , where d ( vi , vj ) is the graph distance between vi and vj . In this paper , we use directed k-hop ego-graph and its direction is decided by whether it is composed of incoming or outgoing edges to the center node , i.e. , gi and g̃i . The results apply trivially to undirected graphs with gi = g̃i . Definition 3.2 ( Structural information ) . Let G be a topological space of sub-graphs , we view a graph G as samples of k-hop ego-graphs { gi } ni=1 drawn i.i.d . from G with probability µ , i.e. , gi i.i.d.∼ µ ∀i = 1 , · · · , n. The structural information of G is then defined to be the set of k-hop ego-graphs of { gi } ni=1 and their empirical distribution . As shown in Figure 1 , three graphs G0 , G1 and G2 are characterized by a set of 1-hop ego-graphs and their empirical distributions , which allows us to quantify the structural similarity among graphs as shown in §3.2 ( i.e. , G0 is more similar to G1 than G2 under such characterization ) . In practice , the nodes in a graph G are characterized not only by their k-hop ego-graph structures but also their associated node features . Therefore , G should be regarded as samples { ( gi , xi ) } drawn from the joint distribution P on the product space of G and a node feature space X . Ego-Graph Information Maximization . Given a set of ego-graphs { ( gi , xi ) } i drawn from an empirical joint distribution ( gi , xi ) ∼ P. We aim to train an GNN encoder Ψ to maximize the mutual informaion ( MI ( gi , Ψ ( gi , xi ) ) ) between the defined structural information gi3 ( i.e . k-hop ego-graph ) and node embedding zi = Ψ ( gi , xi ) . To maximize the MI , another discriminator D ( gi , zi ) : E ( gi ) × zi → R+ is introduced to compute the probability of an edge e belongs to the given ego-graph gi . We use the Jensen-Shannon MI estimator [ 20 ] in the EGI objective , LEGI = −MI ( JSD ) ( G , Ψ ) = 1N N∑ i=1 [ sp ( D ( gi , z′i ) ) + sp ( −D ( gi , zi ) ) ] , ( 1 ) where sp ( x ) = log ( 1+ex ) is the softplus function and ( gi , z′i ) is randomly drawn from the product of marginal distributions , i.e . z′i = Ψ ( gi′ , xi′ ) , ( gi′ , xi′ ) ∼ P , i′ 6= i . In general , we can also randomly draw negative g′i in the topological space , while enumerating all possible graphs gi′ leads to high computation cost . In Eq . 1 , the computation of D on E ( gi ) depends on the node orders . Following the common practice in graph generation [ 70 ] , we characterize the decision process of D with a fixed graph ordering , i.e. , the BFS-ordering π over edges E ( gi ) . D = f ◦ Φ is composed by another GNN encoder Φ and scoring function f over an edge sequence Eπ : { e1 , e2 , ... , en } , which makes predictions on the BFS-ordered edges . 3Later in section 3.2 , we will discuss the equivalence between MI ( gi , zi ) and MI ( ( gi , xi ) , zi ) when node feature is structure-respecting . Recall our previous definition on the direction of k-hop ego-graph , the center node encoder Ψ receives pairs of ( gi , xi ) while the neighbor node encoder Φ in discriminator D receives ( g̃i , xi ) . Both encoders are parameterized as GNNs , Ψ ( gi , xi ) = GNNΨ ( Ai , Xi ) , Φ ( g̃i , xi ) = GNNΦ ( A′i , Xi ) , where Ai , A′i is the adjacency matrix with self-loops of gi and g̃i , respectively . The self-loops are added following the common design of GNNs , which allows the convolutional node embeddings to always incorporate the influence of the center node . Ai = A′i ᵀ . The output of Ψ , i.e. , zi ∈ Rn , is the center node embedding , while Φ outputs representation H ∈ R|gi|×n for neighbor nodes in the ego-graph . Once node representation H is computed , we now describe the scoring function f . For each of the node pair ( p , q ) ∈ Eπ , hp is the source node representation from Φ , xq is the destination node features . The scoring function is , f ( hp , xq , zi ) = σ ( UT · τ ( WT [ hp||xq||zi ] ) ) , ( 2 ) where σ and τ are Sigmoid and ReLU activation functions . Thus , the discriminator D is asked to distinguish a positive ( ( p , q ) , zi ) and negative pair ( ( p , q ) , z′i ) ) for each edge in gi . D ( gi , zi ) = ∑ ( p , q ) ∈Eπ log f ( hp , xq , zi ) , D ( gi , z′i ) = Eπ∑ ( p , q ) log f ( hp , xq , z ′ i ) . ( 3 ) There are two types of edges ( p , q ) in our consideration of node orders , type-a - the edges across different hops ( from the center node ) , and type-b - the edges within the same hop ( from the center node ) . The aforementioned BFS-based node ordering guarantees that Eq . 3 is sensitive to the ordering of type-a edges , and invariant to the ordering of type-b edges , which is consistent with the requirement of our theoretical analysis on ∆D . Due to the fact that the output of a k-layer GNN only depends on a k-hop ego-graph for both encoders Ψ and Φ , EGI can be trained in parallel by sampling batches of gi ’ s . Besides , the training objective of EGI is transferable as long as ( gi , xi ) across source graph Ga and Gb satisfies the conditions given in §3.2 . More model details in Appendix §B and source code in the Supplementary Materials . Connection with existing work . To provide more insights into the EGI objective , we also present it as a dual problem of ego-graph reconstruction . Recall our definition of ego-graph mutual information MI ( gi , Ψ ( gi , xi ) ) . It can be related to an ego-graph reconstruction loss R ( gi|Ψ ( gi , xi ) ) as max MI ( gi , Ψ ( gi , xi ) ) = H ( gi ) −H ( gi|Ψ ( gi , xi ) ) ≤ H ( gi ) −R ( gi|Ψ ( gi , xi ) ) . ( 4 ) When EGI is maximizing the mutual information , it simultaneously minimizes the upper error bound of reconstructing an ego-graph gi . In this view , the key difference between EGI and VGAE [ 28 ] is they assume each edge in a graph to be observed independently during the reconstruction . While in EGI , edges in an ego-graph are observed jointly during the GNN decoding . Moreover , existing mutual information based GNNs such as DGI [ 54 ] and GMI [ 41 ] explicitly measure the mutual information between node features x and GNN output Ψ . In this way , they tend to capture node features instead of graph structures , which we deem more essential in graph transfer learning as discussed in §3.2 . Use cases of EGI framework . In this paper , we focus on the classical domain adaption ( directtransferring ) setting [ 7 ] , where no target domain labels are available and transferability is measured by the performance discrepancy without fine-tuning . In this setting , the transferability of EGI is theoretically guaranteed by Theorem 3.1 . In §4.1 , we validated this with the airport datasets . Beyond direct-transferring , EGI is also useful in the more generalized and practical setting of transfer learning with fine-tuning , which we introduced in §4.2 and validated with the YAGO datasets . In this setting , the transferability of EGI is not rigorously studied yet , but is empirically shown promising . Supportive observations . In the first three columns of our synthetic experimental results ( Table 1 ) , in both cases of transfering GNNs between similar graphs ( F-F ) and dissimilar graphs ( B-F ) , EGI significantly outperforms all competitors when using node degree one-hot encoding as transferable node features . In particular , the performance gains over the untrained GIN show the effectiveness of training and transfering , and our gains are always larger than the two state-of-the-art unsupervised GNNs . Such results clearly indicate advantageous structure preserving capability and transferability of EGI .

Summary: This work aims to provide fundamental understanding towards the mechanism and transferability of GNNs, and develops an unsupervised GNN training objective based on their understanding. Novel theoretical analysis has been done to support the design of EGI and establish its transferability bound, while the effectiveness of EGI and the utility of the transferability bound are verified by extensive experiments. The whole story looks new, comprehensive and convincing to me.