Neo-GNNs: Neighborhood Overlap-aware Graph Neural Networks for Link Prediction

1 Introduction . Graph-structured data is ubiquitous in a wide range of domains ranging from social network analysis [ 1 , 2 , 3 ] to biology [ 4 , 5 , 6 , 7 ] and computer vision [ 8 , 9 , 10 ] . In recent years , numerous variants of Graph neural networks ( GNNs ) have been proposed for learning representations over graph-structured data . GNNs learn low dimensional representations of nodes or graphs via iterative aggregation of features from neighbors using non-linear transformations . In this manner , GNNs have shown significant improvements over traditional methods , e.g. , heuristic methods and embedding-based methods , and achieved state-of-the-art performance on various tasks , such as node classification [ 11 , 12 , 13 , 14 , 15 ] , graph classification [ 16 , 17 , 18 , 19 , 20 ] , and graph generation [ 21 , 22 , 23 , 24 ] . However , in link prediction , traditional heuristic methods still show competitive performance compared to GNNs , which often even outperform GNNs . This is because structural information , ( e.g. , overlapped neighborhoods , degrees , and shortest path ) , is crucial for link prediction whereas GNNs heavily rely on smoothed node features rather than graph structure . Recently , SEAL [ 25 ] has been proposed to consider structural information for link prediction by utilizing the relative distance between the target node pair and their neighborhoods . Nonetheless , SEAL requires the expensive computational cost to apply a GNN independently to an extracted subgraph for each target node pair . To address this limitation , we propose Neighborhood Overlap-aware Graph Neural Networks ( NeoGNNs ) that are designed to consider key structural information regarding links without manual processes . Specifically , instead of using input node features , Neo-GNNs first learn to generate useful structrual features for each node from an adjacency matrix . Then Neo-GNNs measure the existence of links by considering the structural features of overlapped neighbhorhoods via neighbhorhood ∗corresponding author 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . overlap-aware aggregation scheme . Finally , to consider both structural information and input node features , our proposed model adaptively combines scores from Neo-GNNs and feature-based GNNs in an end-to-end fashion . We show that Neo-GNNs consistently outperform both state-of-the-art GNNs and heuristic methods on four Open Graph Benchmark datasets ( OGB ) for link prediction . Furthermore , Our Neo-GNNs generalize the neighborhood overlap-based heuristic methods which measure the likelihood of the link based on manually designed structural information of overlapped neighbors . Our contributions are as follows : ( i ) We propose Neighborhood Overlap-aware Graph Neural Networks ( Neo-GNNs ) that learn useful structural features from an adjacency matrix and estimate overlapped neighborhoods for link prediction . ( ii ) Neo-GNNs generalize neighborhood overlap-based heuristic methods and handle overlapped multi-hop neighborhoods . ( iii ) Our extensive experiments on Open Graph Benchmark datasets ( OGB ) demonstrate that Neo-GNNs consistently achieve state-of-the-art performance in link prediction . 2 Related Works . Graph Neural Networks . GNNs have been designed to learn node representations by using neural networks on graph topology . Among deep learning based approaches , the message passing scheme is dominantly used in recent studies such as GCN [ 11 ] , GraphSAGE [ 26 ] , and GAT [ 12 ] . Due to the iterative aggregation step , each node representation vector can have information of neighbor nodes in multi-hop relationships required for downstream tasks . However , there is a limitation of the expressive power that is upper-bounded by the 1-Weisfeiler-Lehman ( 1-WL ) graph isomorphism test . To overcome this limitation , recent works have tried to boost the expressive power of GNNs by augmenting node features with ordering vectors or position-aware vectors [ 11 , 27 , 28 ] . The main purpose of these works is to complement GNNs with structural information which is crucial for prediction tasks . Our study focuses on adaptively incorporating structural information to GNNs for the link prediction task . Link Prediction . Link prediction has been studied in various ways . Conventionally , diverse heuristic methods have been proposed for link prediction . They basically measure the scores of given node pairs based on structural information e.g. , overlapped neighbors and shortest path , about the pair of nodes . Common neighbors and preferential attachment [ 29 ] exploit structural information about one-hop neighbors to compute the score . To consider more than one-hop relationships , second-order heuristic methods ( e.g. , Adamic-Adar [ 30 ] and resource allocation [ 31 ] ) and higher-order heuristic methods ( e.g. , Katz [ 32 ] , PageRank [ 33 ] , and SimRank [ 34 ] ) have been proposed . Heuristic methods are extremely effective for link prediction . However , they require manually designed structural information for each heuristic method . To overcome this limitation , embedding-based methods have been proposed . They learn node embeddings based on connections between nodes and compute similarity scores using the embeddings . Typically , Matrix factorization [ 35 ] learns node embeddings by decomposing an adjacency matrix of the graph . Random walk-based embedding methods such as Deepwalk [ 36 ] , and node2vec [ 37 ] learn node embeddings by applying the Skip-Gram [ 38 ] techniques on the random walks . LINK [ 39 ] learns to classify the existence of links based on each row in the adjacency matrix , which includes connectivity information . Since the performance of the embedding methods depends on the sparsity of the input graph , it is hard to regard these methods as generalized ones . Recently , with the success of GNNs in learning graph representations , there have been several attempts to apply them to the link prediction task . Typically , GAE and VGAE [ 40 ] learn node representations through GCN to reconstruct the input graph in the auto-encoder framework . Based on the GAE , various GNN architectures have been applied to link prediction . On the other hand , SEAL [ 25 ] reformulated the link prediction task to the classification of enclosing subgraphs . Instead of directly predicting the link , enclosing graphs are sampled around each target link to compose dataset and SEAL performs the graph classification task . Due to the node labeling step to mark nodes ’ different roles in an enclosing subgraph , SEAL has better performance than GAE even though both are GNN-based methods . However , constructing subgraphs is inefficient because it requires a large amount of computation , whereas our model is as efficient as GAE and can consider structural information like SEAL . 3 Methods . The goal of our framework , Neighborhood Overlap-aware Graph Neural Networks ( Neo-GNNs ) , is to learn useful structural features from an adjacency matrix and estimate overlapped neighbors for link prediction . We begin with defining the basic notions of graph neural networks for link prediction and review neighborhood overlap-based heuristic methods , and then introduce Neo-GNNs . 3.1 Preliminaries . Notations . Consider an undirected graph G = ( V , E ) with N nodes , where V = { v1 , v2 , . . . , vN } represents a set of nodes and E = { eij | vi , vj ∈ V } represents a set of edges where the nodes vi , vj ∈ V are connected . The adjacency matrix A ∈ RN×N is defined by Aij = 1 if eij ∈ E and 0 otherwise . The degree matrix D ∈ RN×N is a diagonal matrix defined by Dii = ∑ j Aij . The nodes of G have their own feature vectors xi ∈ RF ( i ∈ { 1 , 2 , . . . , N } ) , with X ∈ RN×F denoting the collection of such vectors in a matrix form . Graph Neural Networks for Link Prediction . Given a graph G and a feature matrix X , graph neural networks learn meaningful node representations by an iterative aggregation of transformed representations of neighbor nodes in each l-th GNN layer as follows : H ( l+1 ) = σ ( ÃGNNH ( l ) W ( l ) ) , ( 1 ) where ÃGNN ∈ RN×N is the adjacency matrix normalized in different ways depending on each GNN architecture ( e.g. , D̃− 1 2 ( A+ I ) D̃− 1 2 ) , W ( l ) ∈ Rd ( l ) ×d ( l+1 ) is a trainable weight matrix , and H ( 0 ) is the node feature matrix X ∈ RN×F . After stacking L GNN layers , node representations H ( L ) are then used to predict existence of each link ( i , j ) : ŷij = σ ( s ( h ( L ) i , h ( L ) j ) ) , ( 2 ) where s ( · , · ) is a function , e.g. , inner product or MLP , and h ( L ) i is the representation of the node i from H ( L ) . 3.2 Neighborhood Overlap-based Heuristic Methods . Heuristic methods for link prediction measure the score of given node pairs based on structural information about the node pairs , e.g. , shortest path , degree , and common neighbors . Although GNNs outperform existing traditional heuristic methods in various graph tasks , in link prediction , since GNNs heavily rely on smoothed node features rather than graph structure , the heuristic methods often show competitive performance compared to GNNs . Especially , neighborhood overlap-based heuristic methods are straightforward yet highly effective , even better than GNN models in several datasets , e.g. , ogbl-collab and ogbl-ppa . Typical neighborhood overlap-based heuristic methods are Common Neighbors , Resource Allocation ( RA ) [ 31 ] , and Adamic Adar [ 30 ] . The Common Neighbors method measures the score of link ( u , v ) by counting the number of common neighbors between node u and v as SCN ( u , v ) = |N ( u ) ∩N ( v ) | = ∑ k∈N ( u ) ∩N ( v ) 1 . ( 3 ) The Common Neighbors method is simple and effective , but has a limitation that equally weighs the importance of each common neighbor . To solve this issue , several heuristic methods e.g. , Resource Allocation , and Adamic-Adar measure the score for the link by considering the importance of each common neighbor . From the intuition that neighbor nodes with lower degrees are more significant , they give more weight to neighbors with lower degrees . Specifically , Resource Allocation ( RA ) [ 31 ] measures the score of link ( u , v ) by counting the inverse degrees of common neighbors between node u and v as SRA ( u , v ) = ∑ k∈N ( u ) ∩N ( v ) 1 dk , ( 4 ) where dk denotes the degree of node k. Adamic-Adar has a relatively decreased penalty for higher degree compared to RA by using the reciprocal logarithm of common neighbors ’ degrees between node u and v as SAA ( u , v ) = ∑ k∈N ( u ) ∩N ( v ) 1 log dk . ( 5 ) These heuristic methods show comparable performance to GNNs for link prediction . However , they have two limitations . First , each heuristic method uses manually designed structural features of neighborhoods , e.g. , 1 , 1d , 1 log d . This requires the manual choice by domain experts to select the best heuristic method for each dataset . Second , they only consider structural similarity . While the GNNs do not use graph structures well compared to using node features , heuristic methods can not utilize the node features for link prediction . To address the limitations of both GNNs and heuristic methods , we propose Neighborhood Overlapaware Graph Neural Networks ( Neo-GNN ) , that learn useful structural features from an adjacency matrix and estimate overlapped neighborhoods for link prediction , and adaptively combine with the conventional feature-based GNNs in an end-to-end fashion .

Summary: The paper proposes to use simple functions learned from the adjacency matrix of a graph to boost the performance of GNNs. This overcomes the problem of using traditional heuristics which are effective but must be computed manually. The proposed approach is simple and the experiments over the quite challenging datasets of the OGB show that it is rather effective. I have some concerns regarding the latter which is why my current score is only slightly below the threshold, but maybe the authors can resolve those.