Adaptive Structural Fingerprints for Graph Attention Networks

1 INTRODUCTION . Many real-world data set are represented naturally as graphs . For example , citation networks specify the citation links among scientific papers ; social media often need to explore the significant amount of connections between users ; biological processes typically involve complex interactions such as protein-protein-interaction ( PPI ) . In these scenarios , the complex structures such as the graph topology or connectivities encode crucial domain-specific knowledge for the learning and prediction tasks . Examples include node embedding or classification , graph classification , and so on . The complexity of graph-structured data makes it non-trivial to employ traditional convolutional neural networks ( CNN ’ s ) . The CNN architecture was originally designed for images whose pixels are located on a uniform grids , and so the convolutional filters can be reused everywhere without having to accommodate local structure changes ( LeCun & Kavukcuoglu , 2010 ) . However , extending CNN to deal with arbitrary structured graphs can be non-trivial . To solve this problem , graph neural networks ( GNN ) were early proposed by Gori et al . ( 2005 ) and Sperduti ( 1997 ) , which adopt ∗Corresponding author †Corresponding author an iterative process to propagate the state of each node , followed by a neural network module to generate the output until an equilibrium state is reached . Recent development of GNN can be categorized into spectral and non-spectral approaches . Spectral approaches employ the tools in signal processing and transform the convolutional operation in the graph domain to much simpler operations of the Laplacian spectrum ( Bruna et al. , 2014 ) , and various approaches have been proposed to localize the convolution in either the graph or spectral domain ( Henaff et al. , 2015 ; Defferrard et al. , 2016 ; Kipf & Welling , 2017 ) . Non-spectral approaches define convolutions directly on the neighboring nodes . As a result , varying node structures have to be accommodated through various processing steps such as fixed-neighborhood size sampling ( Hamilton et al. , 2017 ) , neighborhood normalization ( Niepert et al. , 2016 ) , or learning a weight matrix for each node degree ( Duvenaud et al. , 2015 ) or neighborhood size ( Hamilton et al. , 2017 ) . Recently , residual network is also introduced to graph neural networks ( Zhang & Meng , 2019 ) . Recently , graph attention network ( GAT ) proves a promising framework by combining graph neural networks with attention mechanism ( Velickovic et al. , 2017 ) . The attention mechanism allows dealing with variable sized input while focusing on the most relevant parts , and has been widely used in sequence modelling ( Bahdanau et al. , 2015 ) , machine translation ( Luong et al. , 2015 ) , and visual processing ( Xu et al. , 2015 ) . The GAT model further introduces attention module into graphs , where the hidden representation of the nodes are computed by repeatedly attending over their neighbors ’ features , and the weighting coefficients are calculated inductively based on a self-attention strategy . Despite the numerous success , how to exploit structural information in GAT remains an challenge . Note that attention scores in GAT are computed mainly based on the content of the nodes ; the structures of the graph are simply used to mask the attention , e.g. , only one-hop neighbors will be attended . When considering attention among higher order neighbors , however , the performance of GAT deteriorates ( see experimental section for details ) . This is closely related to over-smoothing of GNN ’ s ( Li et al. , 2018 ) and reflects the weakness of GAT in effectively exploiting graph structural information . In this paper , we believe that the topology or “ shapes ” of local edge connections scan provide a valuable guidance on how to exploit rich structural information of graphs . For example , in social networks or biological networks , a community or pathway may be composed of nodes that are densely inter-connected with each other but several hops away . Therefore , it can be beneficial attend neighbors from the same community , even they show no direct connections . To achieve this , simply checking k-hop neighbors would seem insufficient and a thorough exploration of structural landscapes of the graph becomes necessary . In order to fully exploit rich , high-order structural details in graph attention networks , we propose a new model called “ adaptive structural fingerprints ” . The key idea is to contextualize each node within a local receptive field composed of its high-order neighbors . Each node in the neighborhood will be assigned a non-negative , closed-form weighting based on local information propagation procedures , and so the domain ( or shape ) of the receptive field will adapt automatically to local graph structures and the learning task . We call this weighted , tunable receptive field for each node its “ structural fingerprint ” . The structural fingerprint encodes important structural details and will be used in conjunction with the node feature to compute an improved attention layer . Furthermore , our approach provides a useful platform for different subspaces of the node features and various scales of local graph structures to coordinate with each other in learning multi-head attention , being particularly beneficial in handling complex real-world graph data sets . In Section 2 , we introduce the proposed method , including limitation of content-based attention , construction of adaptive structural fingerprints , and the algorithm workflow . In Section 3 , we discuss related work . Section 4 reports empirical evaluations and the last section concludes the paper . 2 EXPLOITING GRAPH STRUCTURAL DETAILS IN ATTENTION . 2.1 LIMITATIONS OF CONTENT-BASED ATTENTION . The “ closeness ” between two nodes should be determined from both their content and structure . Here we illustrate the importance of detailed graph structures in determining node similarities . In Figure 1 ( a ) , suppose the feature similarity of node-pairs ( A , B ) and ( A , C ) are similar . Namely , content-based attention will be similar for this two node pairs . However , from structural point of view , the attention between ( A , B ) should be much stronger than that for ( A , C ) . This is because A and B are located in a small , densely inter-connected community , and they share a significant portion of common neighbors ; while node A and C does not have any common neighbor . In Figure 1 ( b ) , node A and node B are not direct neighbors , and connecting them takes three edges . As a result , their features will not directly affect each other in the message passing of GAT1 . However , both A and B are strongly connected to a dense community , and node B further connects to the community hub . Therefore , it is reasonable for A and B to directly affect each other . In these examples , feature based similarity alone is insufficient in computing a faithful attention . One may need to take into account structural details of higher-order neighbors and how they interact with each other . In the literature , structural clues have long been exploited in solving problems of clustering , semi-supervised learning , and community detection . For example , normalized cut minimizes edge connections between clusters for node partitioning ( Shi & Malik , 2000 ) ; mean-shift uses density peaks to identify clusters ( Comaniciu & Meer , 2002 ) ; low-density separation ( Chapelle & Zien , 2005 ) assumes that class boundaries pass through low-density regions , leading to successful semi-supervised classification ; in community detection , densely connected subgraphs are the main indicator of community ( Girvan & Newman , 2002 ) . In the following , we show how to build “ adaptive structural fingerprint ” to extract informative structural clues to improve graph attention and henceforth node embedding and classification . 2.2 ADAPTIVE STRUCTURAL FINGERPRINTS . Key to exploiting the structural information is the construction of the so-called ” adaptive structural fingerprints ” , by placing each node in the context of the its local ” receptive field ” . Figure 2 is an illustration . For any node i , consider a spanning process that locates a local subgraph around i , for example , all the nodes within the k-hop neighbors of i. Denote the resultant subgraph as ( Vi , Ei ) , where Vi and Ei are the set of nodes ( dark red ) and edges ( black ) in this neighborhood . In the meantime , each node in i will be assigned a non-negative weight , denoted by wi ∈ Rni×1 , which specifies the importance of each node in shaping the receptive filed also determines the effective “ shape ” and size of the field . The structural fingerprint of node i will be defined as Fi = ( Vi , wi ) . 2.3 CONSTRUCTION OF THE STRUCTURAL FINGERPRINTS . Intuitively , the weight of the nodes should decay with their distance from the center of the fingerprint . A simple idea is to compute the weight of the node j using a Gaussian function of its distance from the center node i , i.e. , wi ( j ) = exp ( −dis ( i , j ) 2 2h2 ) , which we call Gaussian decay . Alternatively , we can map the node distance levels [ 1 , 2 , ... , k ] to weight levels u = [ u1 , u2 , ... , uk ] which are nonnegative and monotonic ; we call nonparametric decay , which is more flexible and will be used in our experiments . In both cases , the decay parameter ( Gaussian bandwidth h or nonparametric decay profile u ) can be optimized , making the structural fingerprint more adaptive to the learning process . In practice , it is more desirable if node weights can be automatically adjusted by local graph structures ( shapes or density of connections ) . To achieve this , we propose to use Random Walk with Restart ( RWR ) . Random walks were first developed to explore global topology of a network by simulating a particle iteratively moving among neighboring nodes ( Lovasz , 1993 ) . If the particle is 1If one considers multiple message passing steps , higher-order neighbors may finally affect each other ; but this can be difficult to analyze and convergence can be slower than interactions within one step . forced to always restart in the same node ( or set of “ seed ” nodes ) , random walk with restart ( RWR ) can then quantify the structural proximity between the seed ( s ) and all the other nodes in the graph , which has been widely used in information retrieval ( Tong et al. , 2006 ; Pan et al. , 2004 ) . Consider a random walk with restart on a structural fingerprint centered around node i , with altogether ni nodes and adjacency matrix Ei . The particle starts from the center node i and randomly walks to its neighbors in Vi with a probability proportional to edge weights . In each step , it also has a certain probability to return to the center node . The iteration can be written as w ( t+1 ) i = c · Ẽiw ( t ) i + ( 1− c ) · ei where Ẽ is the transition probability matrix by normalizing columns of Ei , c ∈ [ 0 , 1 ] is a tradeoff parameter between random walk and restart , and ei is a vector of all zeros except the entry corresponding to the center node i . The converged solution can be written in closed form as wi = ( I− c · Ẽi ) −1ei . ( 1 ) The wi quantifies the proximity between the center node i and all other nodes of the fingerprint , and in the meantime naturally reflects local structural details of the graph . The c controls the decaying rate ( effective size ) of the fingerprint : if c = 0 , wi will all be zeros except the ith node ; if c = 1 , wi will be the stationary distribution of a standard random walk on graph ( Vi , Ei ) . In practice , c will be optimized so that the fingerprint adapts naturally to both the graph structure and the learning task . In Figure 3 , we illustrate Gaussian-based and RWR-based fingerprint weights . We created a toy example of local receptive field , which for convenience is a uniform grid but with a small denser “ patch ” on the right bottom . As can be expected , the Gaussian based weight decay has contours that are center-symmetric . In comparison , the RWR automatically takes into account salient local structures and so the contours will be biased towards the dense subgraph . This is particularly desirable in case the center node is close to ( or residing in ) a community ; it will then be represented more closely by the nodes from community , achieving the effect of a “ structural attractor ” . As can be seen , the “ structural attractor ” effect in building the fingerprint with RWR coincides well with commonly used clustering assumptions , namely , the structural fingerprint of a node will emphasize more on densely inter-connected neighbors within predefined range . Since highly weighted nodes are more likely to come from the same cluster as the center node , the fingerprint of each node is supposed to provide highly informative guidance on its structural identity , thus improving evaluation of node similarites and finally the graph attention .

Summary: This paper extends the idea of self-attention in graph NNs, which is typically based on feature similarity between nodes, to include also structural similarity. This is done by computing for each node a value for each other node within its receptive field, calculated based on some distance metric from the node (either a Gaussian decay profile, or a learned weighting of number of hops distance, of based on fixed point of a random walk with restart). When evaluating the attention between two nodes, a function that compares the structural values between the two nodes (based on Jacard similarity) gives a score to the two nodes, which is further used for calculating the attention weight between them.