Revisiting Graph Neural Networks for Link Prediction

1 INTRODUCTION . Link prediction is to predict potential or missing links connecting pairwise nodes in a network . It has wide applications in various fields , such as friend recommendation in social networks ( Adamic & Adar , 2003 ) , movie recommendation in Netflix ( Bennett et al. , 2007 ) , protein-protein interaction prediction ( Qi et al. , 2006 ) , and knowledge graph completion ( Nickel et al. , 2015 ) , etc . Traditional link prediction approaches include heuristic methods , embedding methods , and featurebased methods . Heuristic methods compute some heuristic node similarity scores as the likelihood of links ( Liben-Nowell & Kleinberg , 2007 ) , such as common neighbors , preferential attachment ( Barabási & Albert , 1999 ) , and Katz index ( Katz , 1953 ) , which can be regarded as some predefined graph structure features . Embedding methods , including matrix factorization ( MF ) and Node2vec ( Grover & Leskovec , 2016 ) , learn free-parameter node embeddings from the observed network transductively , thus do not generalize to unseen nodes and networks . Feature-based methods only use explicit node features yet do not consider the graph structure . Recently , graph neural networks ( GNNs ) emerged to be powerful tools for learning over graph-structured data ( Scarselli et al. , 2009 ; Bruna et al. , 2013 ; Duvenaud et al. , 2015 ; Li et al. , 2015 ; Kipf & Welling , 2016a ; Niepert et al. , 2016 ; Dai et al. , 2016 ) , and have been successfully used in link prediction as well ( Kipf & Welling , 2016b ; Zhang & Chen , 2018 ; You et al. , 2019 ; Chami et al. , 2019 ; Li et al. , 2020 ) . There are two main types of GNN-based link prediction methods . One is Graph Autoencoder ( Kipf & Welling , 2016b ) , where a GNN is first applied to the entire network to learn an embedding vector for each node . Then the embeddings of the source and target nodes are aggregated to predict the target link . The second type is SEAL ( Zhang & Chen , 2018 ; Li et al. , 2020 ) , where an enclosing subgraph is extracted around each target link . Then the nodes in each enclosing subgraph are labeled differently according to their distances to the source and target nodes . Finally a GNN is applied to each enclosing subgraph to learn a link representation for link prediction . At first glance , both methods seem to learn graph structure features associated with the target link , and leverage these structure features for link prediction . However , as we will see , the two methods have fundamentally different power in terms of learning the structural representations of links . 𝑣2 𝑣1 𝑣3 Figure 1 : The structural roles of link ( v1 , v2 ) and link ( v1 , v3 ) are different , but GAE will assign equal probabilities to them . We first show that by individually learning source and target node embeddings , GAE methods can not differentiate links with different structural roles . To intuitively understand this , we give an example in Figure 1 . In this graph , nodes v2 and v3 have the same structural roles ( symmetric/isomorphic to each other ) . A GAE will learn the same node embeddings for v2 and v3 , thus giving the same predicted probabilities for link ( v1 , v2 ) and link ( v1 , v3 ) . However , the structural roles of link ( v1 , v2 ) and link ( v1 , v3 ) are apparently different – v1 intuitively should have unequal probabil- ities connecting to v2 and v3 . Next , we propose a labeling trick , which gives a label to each node as its additional feature , where the source and target nodes are labeled differently from the rest . We show that combined with the labeling trick , a sufficiently expressive GNN can learn the same representations for two links if and only if their structural roles are the same within the graph . This way , ( v1 , v2 ) and ( v1 , v3 ) will be predicted differently in Figure 1 . We further show that SEAL is such an example . Finally , we give a more practical definition of isomorphism , called local isomorphism , which defines two nodes/links as isomorphic if their local neighborhood subgraphs are isomorphic . We argue that GNNs for link prediction should target on local-isomorphism-discriminating . We conduct a thorough comparison among different link prediction methods , including SEAL and various GAE and embedding methods , on the recent large-scale Open Graph Benchmark ( OGB ) datasets ( Hu et al. , 2020 ) . We show that SEAL with the labeling trick has up to 195 % higher Hits @ 100 than GAE methods , achieving new state-of-the-art results on 3 out of 4 datasets . 2 PRELIMINARIES . In this section , we formally define the notions of graph , permutation , isomorphism , and GNN . Definition 1 . ( Graph ) . We consider an undirected graph G = ( V , E , A ) , where V = { 1 , 2 , . . . , n } is the set of n vertices , E ⊆ V ×V is the set of edges , and A ∈ Rn×n×k contains the node and edge features with its diagonal components Ai , i , : denoting node attributes and off-diagonal components Ai , j , : denoting edge attributes . We further use A ∈ { 0 , 1 } n×n to denote the adjacency matrix of G with Ai , j = 1 iff ( i , j ) ∈ E. If there are no node/edge features , we let A = A . Otherwise , A can be regarded as the first slice of A , i.e. , A = A : , : ,1 . Definition 2 . ( Permutation ) A node permutation π is a bijective mapping from { 1 , 2 , . . . , n } to { 1 , 2 , . . . , n } . All n ! possible π ’ s constitute the permutation group Πn . We define π ( S ) = { π ( i ) |i ∈ S } when S is a subset of { 1 , 2 , . . . , n } . We further define the permutation of A as π ( A ) , where π ( A ) π ( i ) , π ( j ) , : = Ai , j , : . In other words , π ( A ) i , j , : = Aπ−1 ( i ) , π−1 ( j ) , : . Definition 3 . ( Set isomorphism ) Given two n-node graphs G = ( V , E , A ) , G′ = ( V ′ , E′ , A′ ) , and two node sets S ⊆ V , S′ ⊆ V ′ , we say ( S , A ) and ( S′ , A′ ) are isomorphic ( denoted by ( S , A ) ' ( S′ , A′ ) ) if ∃π ∈ Πn such that S = π ( S′ ) and A = π ( A′ ) . When ( V , A ) ' ( V ′ , A′ ) , we say two graphs G and G′ are isomorphic ( abbreviated as A ' A′ because V = π ( V ′ ) for any π ) . Note that set isomorphism is more strict than graph isomorphism , because it not only requires graph isomorphism , but also requires the permutation maps a specific subset S to another subset S′ . When S ⊂ V and S′ ⊂ V ′ , we are often more concerned with the case of A = A′ , where we are to find isomorphic node sets in the same graph ( automorphism ) . For example , when S = { i } , S′ = { j } ( single node case ) and ( i , A ) , ( j , A ) are isomorphic , it means i and j are on the same orbit of graph A ( i.e. , they have symmetric positions/same structural roles within the graph ) . An example is v2 and v3 in Figure 1 . Definition 4 . ( Invariant function ) A function f defined over the space of ( S , A ) is invariant if ∀π ∈ Πn , f ( S , A ) = f ( π ( S ) , π ( A ) ) . Definition 5 . ( GNN ) A GNN is an invariant function mapping from the space of ( S , A ) to Rd . More specifically , a GNN first performs multiple invariant message passing operations to compute a node embedding zi = GNN ( i , A ) for all i ∈ S , and then performs a set aggregation ( pooling ) over { zi|i ∈ S } , written as AGG ( { zi|i ∈ S } ) , as the set S ’ s representation GNN ( S , A ) . Note that , when |S| = 1 , the set aggregation is often an identity mapping . In graph classification ( S = V ) , we use a graph pooling layer over node embeddings to compute the graph representation . 3 GAE AND STRUCTURAL LINK REPRESENTATION . In this section , we review how GAE methods predict links , and show that simply aggregating node embeddings learned by a GNN can not lead to effective link representations . We use A to denote the incomplete network to perform link prediction . 3.1 GAE FOR LINK PREDICTION . Graph Autoencoder ( GAE ) methods ( Kipf & Welling , 2016b ) first use a GNN to compute a node embedding zi for each node i , and then use f ( zi , zj ) to predict the link ( i , j ) : Âi , j = f ( zi , zj ) , where zi = GNN ( i , A ) , zj = GNN ( j , A ) ( 1 ) where Âi , j is the predicted score for link ( i , j ) . The model is trained to maximize the likelihood of reconstructing the true adjacency matrix . The original GAE uses a two-layer GCN ( Kipf & Welling , 2016a ) as the GNN , and let f ( zi , zj ) : = σ ( z > i zj ) . In principle , we can replace GCN with any message passing neural network ( Gilmer et al. , 2017 ) , and use an MLP over the aggregation of zi and zj as the f ( zi , zj ) . Popular aggregation functions include concatenation , mean and Hadamard product , etc . In the following , we will use GAE to denote a general class of GNN-based link prediction methods , without differentiating the specific choices of GNN and f . 3.2 GAE CAN LEARN STRUCTURAL NODE REPRESENTATIONS . Following ( Srinivasan & Ribeiro , 2020 ; Li et al. , 2020 ) , we first define most expressive structural representations for nodes and links . Then we relate them to GAE-learned node embeddings and show that GAE is not capable of learning structural link representations . Definition 6 . Given an invariant function Γ ( · ) , Γ ( S , A ) is a most expressive structural representation for ( S , A ) if ∀ ( S , A , S′ , A′ ) , Γ ( S , A ) = Γ ( S′ , A′ ) ⇔ ( S , A ) ' ( S′ , A′ ) . For simplicity , we will briefly use “ structural representation ” to denote most expressive structural representation in the rest of the paper . We will omit A if it is clear from context . We call Γ ( i , A ) a structural node representation for i , and call Γ ( { i , j } , A ) a structural link representation for ( i , j ) . The above definition indicates that two node sets have the same structural representations if and only if they are isomorphic to each other . In the same graph A , structural representations uniquely mark the structural roles of nodes or node sets . This is in contrast to positional node embeddings such as DeepWalk ( Perozzi et al. , 2014 ) and matrix factorization ( Mnih & Salakhutdinov , 2008 ) , where two isomorphic nodes can have different node embeddings ( Ribeiro et al. , 2017 ) . So why do we need to define structural representations ? From a node classification point of view , it is because two isomorphic nodes in a network are perfectly symmetric to each other , and should be indistinguishable using any labeling functions on graphs ( i.e. , they should have the same ground truth y ) . Learning a structural node representation can guarantee that isomorphic nodes are always classified into the same class . Then , a natural question to ask is , do GNNs learn structural node representations ? The answer is no . Recall that ( i , A ) ' ( j , A′ ) ⇒ A ' A′ . If a GNN can learn structural node representations , we can always use it for graph isomorphism test by checking whether there exist two nodes in two graphs sharing the same structural node representation . In fact , existing GNNs ’ graph discriminating power is bounded by the Weisfeiler-Lehman ( WL ) test ( Morris et al. , 2019 ; Maron et al. , 2019 ) , which provably fails to distinguish certain non-isomorphic graphs ( Cai et al. , 1992 ) . Despite this , GNNs/WL are still powerful enough to learn representations that can distinguish almost all nonisomorphic nodes and graphs ( Babai & Kucera , 1979 ) . For easy analysis , we assume there exists a node-most-expressive GNN that can output structural node representations thus able to distinguish all non-isomorphic nodes . Despite this , the techniques we will present are not limited to node-most-expressive GNNs , but also benefit practical GNNs . Definition 7 . A GNN is node-most-expressive if ∀ ( i , A , j , A′ ) , GNN ( i , A ) = GNN ( j , A′ ) ⇔ ( i , A ) ' ( j , A′ ) . Recall that GAE first uses a GNN to compute node embeddings . Therefore , GAE with a node-mostexpressive GNN is able to leverage structural node representations for link prediction . 3.3 GAE CAN NOT LEARN STRUCTURAL LINK REPRESENTATIONS The next question to ask is whether GAE learns structural link representations . That is , does the aggregation of structural node representations of i and j result in a structural link representation of ( i , j ) ? The answer is no , as shown in previous works ( Srinivasan & Ribeiro , 2020 ; Zhang & Chen , 2020 ) . We have also illustrated it in the introduction . In Figure 1 , we have two isomorphic nodes v2 and v3 , thus v2 and v3 will have the same structural node representation . By aggregating structural node representations as link representations , GAE will assign ( v1 , v2 ) and ( v1 , v3 ) the same link representation and predict them to have equal probabilities of forming a link . However , ( v1 , v2 ) and ( v1 , v3 ) apparently have different structural link representations , which indicates that Proposition 1 . Even with a node-most-expressive GNN , GAE can not learn structural link representations . The root cause of this problem is that GAE learns representations for the source and target nodes individually , without considering their relative positions and associations . For example , although v2 and v3 are perfectly symmetric in the graph , when considering the source node v1 to predict link from , v2 and v3 ’ s positions w.r.t . v1 are no longer symmetric .

Summary: This paper provides theoretical analysis of graph neural networks for link prediction, following a number of recent papers that have developed the theoretical understanding of graph neural networks. For example, the work of Xu et al (ICLR 2019) draws on the Weisfeiler-Lehman graph isomorphism test to develop a theoretical framework in which to analyse the expressive power of Graph Neural Networks. In particular, they show that architectural features in common graph neural network models limit the expressivity with respect to problems of node and graph classification and show how one can construct graph neural networks whose expressivity matches the Weisfeiler-Lehmam test. In more recent work, Li et al (NeurIPS 2020) show that augmenting node features with a "distance encoding" enables a graph neural network to distinguish node sets in cases where the Weisfeiler-Lehman test fails. This submission is perhaps most closely related to this recent paper of Li et al.