Neo-GNNs: Neighborhood Overlap-aware Graph Neural Networks for Link Prediction

1 Introduction . Graph-structured data is ubiquitous in a wide range of domains ranging from social network analysis [ 1 , 2 , 3 ] to biology [ 4 , 5 , 6 , 7 ] and computer vision [ 8 , 9 , 10 ] . In recent years , numerous variants of Graph neural networks ( GNNs ) have been proposed for learning representations over graph-structured data . GNNs learn low dimensional representations of nodes or graphs via iterative aggregation of features from neighbors using non-linear transformations . In this manner , GNNs have shown significant improvements over traditional methods , e.g. , heuristic methods and embedding-based methods , and achieved state-of-the-art performance on various tasks , such as node classification [ 11 , 12 , 13 , 14 , 15 ] , graph classification [ 16 , 17 , 18 , 19 , 20 ] , and graph generation [ 21 , 22 , 23 , 24 ] . However , in link prediction , traditional heuristic methods still show competitive performance compared to GNNs , which often even outperform GNNs . This is because structural information , ( e.g. , overlapped neighborhoods , degrees , and shortest path ) , is crucial for link prediction whereas GNNs heavily rely on smoothed node features rather than graph structure . Recently , SEAL [ 25 ] has been proposed to consider structural information for link prediction by utilizing the relative distance between the target node pair and their neighborhoods . Nonetheless , SEAL requires the expensive computational cost to apply a GNN independently to an extracted subgraph for each target node pair . To address this limitation , we propose Neighborhood Overlap-aware Graph Neural Networks ( NeoGNNs ) that are designed to consider key structural information regarding links without manual processes . Specifically , instead of using input node features , Neo-GNNs first learn to generate useful structrual features for each node from an adjacency matrix . Then Neo-GNNs measure the existence of links by considering the structural features of overlapped neighbhorhoods via neighbhorhood ∗corresponding author 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . overlap-aware aggregation scheme . Finally , to consider both structural information and input node features , our proposed model adaptively combines scores from Neo-GNNs and feature-based GNNs in an end-to-end fashion . We show that Neo-GNNs consistently outperform both state-of-the-art GNNs and heuristic methods on four Open Graph Benchmark datasets ( OGB ) for link prediction . Furthermore , Our Neo-GNNs generalize the neighborhood overlap-based heuristic methods which measure the likelihood of the link based on manually designed structural information of overlapped neighbors . Our contributions are as follows : ( i ) We propose Neighborhood Overlap-aware Graph Neural Networks ( Neo-GNNs ) that learn useful structural features from an adjacency matrix and estimate overlapped neighborhoods for link prediction . ( ii ) Neo-GNNs generalize neighborhood overlap-based heuristic methods and handle overlapped multi-hop neighborhoods . ( iii ) Our extensive experiments on Open Graph Benchmark datasets ( OGB ) demonstrate that Neo-GNNs consistently achieve state-of-the-art performance in link prediction . 2 Related Works . Graph Neural Networks . GNNs have been designed to learn node representations by using neural networks on graph topology . Among deep learning based approaches , the message passing scheme is dominantly used in recent studies such as GCN [ 11 ] , GraphSAGE [ 26 ] , and GAT [ 12 ] . Due to the iterative aggregation step , each node representation vector can have information of neighbor nodes in multi-hop relationships required for downstream tasks . However , there is a limitation of the expressive power that is upper-bounded by the 1-Weisfeiler-Lehman ( 1-WL ) graph isomorphism test . To overcome this limitation , recent works have tried to boost the expressive power of GNNs by augmenting node features with ordering vectors or position-aware vectors [ 11 , 27 , 28 ] . The main purpose of these works is to complement GNNs with structural information which is crucial for prediction tasks . Our study focuses on adaptively incorporating structural information to GNNs for the link prediction task . Link Prediction . Link prediction has been studied in various ways . Conventionally , diverse heuristic methods have been proposed for link prediction . They basically measure the scores of given node pairs based on structural information e.g. , overlapped neighbors and shortest path , about the pair of nodes . Common neighbors and preferential attachment [ 29 ] exploit structural information about one-hop neighbors to compute the score . To consider more than one-hop relationships , second-order heuristic methods ( e.g. , Adamic-Adar [ 30 ] and resource allocation [ 31 ] ) and higher-order heuristic methods ( e.g. , Katz [ 32 ] , PageRank [ 33 ] , and SimRank [ 34 ] ) have been proposed . Heuristic methods are extremely effective for link prediction . However , they require manually designed structural information for each heuristic method . To overcome this limitation , embedding-based methods have been proposed . They learn node embeddings based on connections between nodes and compute similarity scores using the embeddings . Typically , Matrix factorization [ 35 ] learns node embeddings by decomposing an adjacency matrix of the graph . Random walk-based embedding methods such as Deepwalk [ 36 ] , and node2vec [ 37 ] learn node embeddings by applying the Skip-Gram [ 38 ] techniques on the random walks . LINK [ 39 ] learns to classify the existence of links based on each row in the adjacency matrix , which includes connectivity information . Since the performance of the embedding methods depends on the sparsity of the input graph , it is hard to regard these methods as generalized ones . Recently , with the success of GNNs in learning graph representations , there have been several attempts to apply them to the link prediction task . Typically , GAE and VGAE [ 40 ] learn node representations through GCN to reconstruct the input graph in the auto-encoder framework . Based on the GAE , various GNN architectures have been applied to link prediction . On the other hand , SEAL [ 25 ] reformulated the link prediction task to the classification of enclosing subgraphs . Instead of directly predicting the link , enclosing graphs are sampled around each target link to compose dataset and SEAL performs the graph classification task . Due to the node labeling step to mark nodes ’ different roles in an enclosing subgraph , SEAL has better performance than GAE even though both are GNN-based methods . However , constructing subgraphs is inefficient because it requires a large amount of computation , whereas our model is as efficient as GAE and can consider structural information like SEAL . 3 Methods . The goal of our framework , Neighborhood Overlap-aware Graph Neural Networks ( Neo-GNNs ) , is to learn useful structural features from an adjacency matrix and estimate overlapped neighbors for link prediction . We begin with defining the basic notions of graph neural networks for link prediction and review neighborhood overlap-based heuristic methods , and then introduce Neo-GNNs . 3.1 Preliminaries . Notations . Consider an undirected graph G = ( V , E ) with N nodes , where V = { v1 , v2 , . . . , vN } represents a set of nodes and E = { eij | vi , vj ∈ V } represents a set of edges where the nodes vi , vj ∈ V are connected . The adjacency matrix A ∈ RN×N is defined by Aij = 1 if eij ∈ E and 0 otherwise . The degree matrix D ∈ RN×N is a diagonal matrix defined by Dii = ∑ j Aij . The nodes of G have their own feature vectors xi ∈ RF ( i ∈ { 1 , 2 , . . . , N } ) , with X ∈ RN×F denoting the collection of such vectors in a matrix form . Graph Neural Networks for Link Prediction . Given a graph G and a feature matrix X , graph neural networks learn meaningful node representations by an iterative aggregation of transformed representations of neighbor nodes in each l-th GNN layer as follows : H ( l+1 ) = σ ( ÃGNNH ( l ) W ( l ) ) , ( 1 ) where ÃGNN ∈ RN×N is the adjacency matrix normalized in different ways depending on each GNN architecture ( e.g. , D̃− 1 2 ( A+ I ) D̃− 1 2 ) , W ( l ) ∈ Rd ( l ) ×d ( l+1 ) is a trainable weight matrix , and H ( 0 ) is the node feature matrix X ∈ RN×F . After stacking L GNN layers , node representations H ( L ) are then used to predict existence of each link ( i , j ) : ŷij = σ ( s ( h ( L ) i , h ( L ) j ) ) , ( 2 ) where s ( · , · ) is a function , e.g. , inner product or MLP , and h ( L ) i is the representation of the node i from H ( L ) . 3.2 Neighborhood Overlap-based Heuristic Methods . Heuristic methods for link prediction measure the score of given node pairs based on structural information about the node pairs , e.g. , shortest path , degree , and common neighbors . Although GNNs outperform existing traditional heuristic methods in various graph tasks , in link prediction , since GNNs heavily rely on smoothed node features rather than graph structure , the heuristic methods often show competitive performance compared to GNNs . Especially , neighborhood overlap-based heuristic methods are straightforward yet highly effective , even better than GNN models in several datasets , e.g. , ogbl-collab and ogbl-ppa . Typical neighborhood overlap-based heuristic methods are Common Neighbors , Resource Allocation ( RA ) [ 31 ] , and Adamic Adar [ 30 ] . The Common Neighbors method measures the score of link ( u , v ) by counting the number of common neighbors between node u and v as SCN ( u , v ) = |N ( u ) ∩N ( v ) | = ∑ k∈N ( u ) ∩N ( v ) 1 . ( 3 ) The Common Neighbors method is simple and effective , but has a limitation that equally weighs the importance of each common neighbor . To solve this issue , several heuristic methods e.g. , Resource Allocation , and Adamic-Adar measure the score for the link by considering the importance of each common neighbor . From the intuition that neighbor nodes with lower degrees are more significant , they give more weight to neighbors with lower degrees . Specifically , Resource Allocation ( RA ) [ 31 ] measures the score of link ( u , v ) by counting the inverse degrees of common neighbors between node u and v as SRA ( u , v ) = ∑ k∈N ( u ) ∩N ( v ) 1 dk , ( 4 ) where dk denotes the degree of node k. Adamic-Adar has a relatively decreased penalty for higher degree compared to RA by using the reciprocal logarithm of common neighbors ’ degrees between node u and v as SAA ( u , v ) = ∑ k∈N ( u ) ∩N ( v ) 1 log dk . ( 5 ) These heuristic methods show comparable performance to GNNs for link prediction . However , they have two limitations . First , each heuristic method uses manually designed structural features of neighborhoods , e.g. , 1 , 1d , 1 log d . This requires the manual choice by domain experts to select the best heuristic method for each dataset . Second , they only consider structural similarity . While the GNNs do not use graph structures well compared to using node features , heuristic methods can not utilize the node features for link prediction . To address the limitations of both GNNs and heuristic methods , we propose Neighborhood Overlapaware Graph Neural Networks ( Neo-GNN ) , that learn useful structural features from an adjacency matrix and estimate overlapped neighborhoods for link prediction , and adaptively combine with the conventional feature-based GNNs in an end-to-end fashion .

Summary: This work added a model into GNN to improve link prediction. The connection is a weighted combination of the model's output and GNN's output in the scoring function (scoring a predicted link).  The paper claims that this model computes the number of overlapped neighbors. Experiments show the combination improves the performance.