Meta-Graph: Few shot Link Prediction via Meta Learning

1 INTRODUCTION . Given a graph representing known relationships between a set of nodes , the goal of link prediction is to learn from the graph and infer novel or previously unknown relationships ( Liben-Nowell & Kleinberg , 2003 ) . For instance , in a social network we may use link prediction to power a friendship recommendation system ( Aiello et al. , 2012 ) , or in the case of biological network data we might use link prediction to infer possible relationships between drugs , proteins , and diseases ( Zitnik & Leskovec , 2017 ) . However , despite its popularity , previous work on link prediction generally focuses only on one particular problem setting : it generally assumes that link prediction is to be performed on a single large graph and that this graph is relatively complete , i.e. , that at least 50 % of the true edges are observed during training ( e.g. , see Grover & Leskovec , 2016 ; Kipf & Welling , 2016b ; Liben-Nowell & Kleinberg , 2003 ; Lü & Zhou , 2011 ) . In this work , we consider the more challenging setting of few shot link prediction , where the goal is to perform link prediction on multiple graphs that contain only a small fraction of their true , underlying edges . This task is inspired by applications where we have access to multiple graphs from a single domain but where each of these individual graphs contains only a small fraction of the true , underlying edges . For example , in the biological setting , high-throughput interactomics offers the possibility to estimate thousands of biological interaction networks from different tissues , cell types , and organisms ( Barrios-Rodiles et al. , 2005 ) ; however , these estimated relationships can be noisy and sparse , and we need learning algorithms that can leverage information across these multiple graphs in order to overcome this sparsity . Similarly , in the e-commerce and social network settings , link prediction can often have a large impact in cases where we must quickly make predictions on sparsely-estimated graphs , such as when a service has been recently deployed to a new locale . That is to say to link prediction for a new sparse graph can benefit from transferring knowledge from other , possibly more dense , graphs assuming there is exploitable shared structure . We term this problem of link prediction from sparsely-estimated multi-graph data as few shot link prediction analogous to the popular few shot classification setting ( Miller et al. , 2000 ; Lake et al. , 2011 ; Koch et al. , 2015 ) . The goal of few shot link prediction is to observe many examples of graphs from a particular domain and leverage this experience to enable fast adaptation and higher accuracy when predicting edges on a new , sparsely-estimated graph from the same domain—a task that can can also be viewed as a form of meta learning , or learning to learn ( Bengio et al. , 1990 ; 1992 ; Thrun & Pratt , 2012 ; Schmidhuber , 1987 ) in the context of link prediction . This few shot link prediction setting is particularly challenging as current link prediction methods are generally ill-equipped to transfer knowledge between graphs in a multi-graph setting and are also unable to effectively learn from very sparse data . Present work . We introduce a new framework called Meta-Graph for few shot link prediction and also introduce a series of benchmarks for this task . We adapt the classical gradient-based metalearning formulation for few shot classification ( Miller et al. , 2000 ; Lake et al. , 2011 ; Koch et al. , 2015 ) to the graph domain . Specifically , we consider a distribution over graphs as the distribution over tasks from which a global set of parameters are learnt , and we deploy this strategy to train graph neural networks ( GNNs ) that are capable of few-shot link prediction . To further bootstrap fast adaptation to new graphs we also introduce a graph signature function , which learns how to map the structure of an input graph to an effective initialization point for a GNN link prediction model . We experimentally validate our approach on three link prediction benchmarks . We find that our MetaGraph approach not only achieves fast adaptation but also converges to a better overall solution in many experimental settings , with an average improvement of 5.3 % in AUC at convergence over non-meta learning baselines . 2 PRELIMINARIES AND PROBLEM DEFINITION . The basic set-up for few shot link prediction is as follows : We assume that we have a distribution p ( G ) over graphs , from which we can sample training graphs Gi ∼ p ( G ) , where each Gi = ( Vi , Ei , Xi ) is defined by a set of nodes Vi , edges Ei , and matrix of real-valued node attributes X ∈ R|Vi|×d . When convenient , we will also equivalently represent a graph as Gi = ( Vi , Ai , Xi ) , where Ai ∈ Z|Vi|×|Vi| is an adjacency matrix representation of the edges in Ei . We assume that each of these sampled graphs , Gi , is a simple graph ( i.e. , contain a single type of relation and no self loops ) and that every node v ∈ Vi in the graph is associated with a real valued attribute vector xv ∈ Rd from a common vector space . We further assume that for each graph Gi we have access to only a sparse subset of the true edges E traini ⊂ Ei ( with |E traini | < < |Ei| ) during training . In terms of distributional assumptions we assume that this p ( G ) is defined over a set of related graphs ( e.g. , graphs drawn from a common domain or application setting ) . Our goal is to learn a global or meta link prediction model from a set of sampled training graphs Gi ∼ p ( G ) , i = 1 ... n , such that we can use this meta model to quickly learn an effective link prediction model on a newly sampled graph G∗ ∼ p ( G ) . More specifically , we wish to optimize a global set of parameters θ , as well as a graph signature function ψ ( Gi ) , which can be used together to generate an effective parameter initialization , φi , for a local link prediction model on graph Gi . Relationship to standard link prediction . Few shot link prediction differs from standard link prediction in three important ways : 1 . Rather than learning from a single graph G , we are learning from multiple graphs { G1 , ... , Gn } sampled from a common distribution or domain . 2 . We presume access to only a very sparse sample of true edges . Concretely , we focus on settings where at most 30 % of the edges in Ei are observed during training , i.e. , where |E train| |E| ≤ 0.3 . 1 1By “ true edges ” we mean the full set of ground truth edges available in a particular dataset . 3 . We distinguish between the global parameters , which are used to encode knowledge about the underlying distribution of graphs , and the local parameters φi , which are optimized to perform link prediction on a specific graph Gi . This distinction allows us to consider leveraging information from multiple graphs , while still allowing for individually-tuned link prediction models on each specific graph . Relationship to traditional meta learning . Traditional meta learning for few-shot classification , generally assumes a distribution p ( T ) over classification tasks , with the goal of learning global parameters that can facilitate fast adaptation to a newly sampled task Ti ∼ p ( T ) with few examples . We instead consider a distribution p ( G ) over graphs with the goal of performing link prediction on a newly sampled graph . An important complication of this graph setting is that the individual predictions for each graph ( i.e. , the training edges ) are not i.i.d .. Furthermore , for few shot link prediction we require training samples as a sparse subset of true edges that represents a small percentage of all edges in a graph . Note that for very small percentages we effectively break all graph structure and recover the supervised setting for few shot classification and thus simplifying the problem . 3 PROPOSED APPROACH . We now outline our proposed approach , Meta-Graph , to the few shot link prediction problem . We first describe how we define the local link prediction models , which are used to perform link prediction on each specific graph Gi . Next , we discuss our novel gradient-based meta learning approach to define a global model that can learn from multiple graphs to generate effective parameter initializations for the local models . The key idea behind Meta-Graph is that we use gradient-based meta learning to optimize a shared parameter initialization θ for the local models , while also learning a parametric encoding of each graph Gi that can be used to modulate this parameter initialization in a graph-specific way ( Figure 1 ) . 3.1 LOCAL LINK PREDICTION MODEL . In principle , our framework can be combined with a wide variety of GNN-based link prediction approaches , but here we focus on variational graph autoencoders ( VGAEs ) ( Kipf & Welling , 2016b ) as our base link prediction framework . Formally , given a graph G = ( V , A , X ) , the VGAE learns an inference model , qφ , that defines a distribution over node embeddings qφ ( Z|A , X ) , where each row zv ∈ Rd of Z ∈ R|V|×d is a node embedding that can be used to score the likelihood of an edge existing between pairs of nodes . The parameters of the inference model are shared across all the nodes in G , to define the approximate posterior qφ ( zv|A , X ) = N ( zv|µv , diag ( σ2v ) ) , where the parameters of the normal distribution are learned via GNNs : µ = GNNµ ( A , X ) , and log ( σ ) = GNNσ ( A , X ) . ( 1 ) The generative component of the VGAE is then defined as p ( A|Z ) = N∏ i=1 N∏ j=1 p ( Au , v|zu , zv ) , with p ( Au , v|zu , zv ) = σ ( z > u zv ) , ( 2 ) i.e. , the likelihood of an edge existing between two nodes , u and v , is proportional to the dot product of their node embeddings . Given the above components , the inference GNNs can be trained to minimize the variational lower bound on the training data : LG = Eqφ [ log p ( Atrain|Z ) ] −KL [ qφ ( Z|X , Atrain ) ||p ( z ) ] , ( 3 ) where a Gaussian prior is used for p ( z ) . We build upon VGAEs due to their strong performance on standard link prediction benchmarks ( Kipf & Welling , 2016b ) , as well as the fact that they have a well-defined probabilistic interpretation that generalizes many embedding-based approaches to link prediction ( e.g. , node2vec ( Grover & Leskovec , 2016 ) ) . We describe the specific GNN implementations we deploy for the inference model in Section 3.3 . 3.2 OVERVIEW OF META-GRAPH . The key idea behind Meta-Graph is that we use gradient-based meta learning to optimize a shared parameter initialization θ for the inference models of a VGAE , while also learning a parametric encoding ψ ( Gi ) that modulates this parameter initialization in a graph-specific way . Specifically , given a sampled training graph Gi , we initialize the inference model qφi for a VGAE link prediction model using a combination of two learned components : • A global initialization , θ , that is used to initialize all the parameters of the GNNs in the inference model . The global parameters θ are optimized via second-order gradient descent to provide an effective initialization point for any graph sampled from the distribution p ( G ) . • A graph signature sGi = ψ ( Gi ) that is used to modulate the parameters of inference model φi based on the history of observed training graphs . In particular , we assume that the inference model qφi for each graph Gi can be conditioned on the graph signature . That is , we augment the inference model to qφi ( Z|A , X , sGi ) , where we also include the graph signature sGi as a conditioning input . We use a k-layer graph convolutional network ( GCN ) ( Kipf & Welling , 2016a ) , with sum pooling to compute the signature : sG = ψ ( G ) = MLP ( ∑ v∈V zv ) with Z = GCN ( A , X ) , ( 4 ) where GCN denotes a k-layer GCN ( as defined in ( Kipf & Welling , 2016a ) ) , MLP denotes a densely-connected neural network , and we are summing over the node embeddings zv output from the GCN . As with the global parameters θ , the graph signature model ψ is optimized via second-order gradient descent . The overall Meta-Graph architecture is detailed in Figure 1 and the core learning algorithm is summarized in the algorithm block below . Algorithm 1 : Meta-Graph for Few Shot Link Prediction Result : Global parameters θ , Graph signature function ψ Initialize learning rates : α , Sample a mini-batch of graphs , Gbatch from p ( G ) ; for each G ∈ Gbatch do E = E train ∪ Eval ∪ E test // Split edges into train , val , and test sG = ψ ( G , E train ) // Compute graph signature Initialize : φ ( 0 ) ← θ // Initialize local parameters via global parameters for k in [ 1 : K ] do sG = stopgrad ( sG ) // Stop Gradients to Graph Signature Ltrain = Eq [ log p ( Atrain|Z ) ] −KL [ qφ ( Z|E train , sG ) ||p ( z ) ] Update φ ( k ) ← φ ( k−1 ) − α∇φLtrain end Initialize : θ ← φK sG = ψ ( G , Eval ∪ E train ) // Compute graph signature with validation edges Lval = Eq [ log p ( Aval|Z ) ] −KL [ q ( Z|Eval ∪ E train , sG ) ||p ( z ) ] Update θ ← θ − ∇θLval Update ψ ← ψ − ∇ψLval end The basic idea behind the algorithm is that we ( i ) sample a batch of training graphs , ( ii ) initialize VGAE link prediction models for these training graphs using our global parameters and signature function , ( iii ) run K steps of gradient descent to optimize each of these VGAE models , and ( iv ) use second order gradient descent to update the global parameters and signature function based on a held-out validation set of edges . As depicted in Fig 1 , this corresponds to updating the GCN based encoder for the local link prediction parameters φj and global parameters θ along with the graph signature function ψ using second order gradients . Note that since we are running K steps of gradient descent within the inner loop of Algorithm 1 , we are also “ meta ” optimizing for fast adaptation , as θ and ψ are being trained via second-order gradient descent to optimize the local model performance after K gradient updates , where generally K ∈ { 0 , 1 , . . . , 5 } .

Summary: Overview: In this paper, a meta-learning approach is proposed to perform link prediction across multi-graphs with scarce data. To do so, each graph is treated as a link prediction "task". Different from the tasks in conventional meta-learning, the graphs here are generally non i.i.d. Based on the variational graph auto-coders, the method learns two important components: the global parameters used for GNNs and a graph signature function used to modulate the parameters of inference model based on the history of observed training graphs.