Multi-hop Attention Graph Neural Network

1 INTRODUCTION . The introduction of the self-attention mechanism ( Bahdanau et al. , 2015 ; Vaswani et al. , 2017 ) , has pushed the state-of-the-art in many domains including graph presentation learning ( Radford et al. , 2019 ; Devlin et al. , 2019 ; Liu et al. , 2019a ; Lan et al. , 2019 ) . Graph Attention Network ( GAT ) ( Veličković et al. , 2018 ) and related models ( Li et al. , 2018 ; Wang et al. , 2019a ; Liu et al. , 2019b ; Oono & Suzuki , 2020 ) developed attention mechanism for Graph Neural Networks ( GNNs ) , which compute attention scores between nodes connected by an edge , allowing the model to attend to messages of node ’ s direct neighbors according to their attention scores . However , such attention computation on pairs of nodes connected by edges implies that a node can only attend to its immediate neighbors to compute its ( next layer ) representation . This implies that receptive field of a single GNN layer is restricted to one-hop network neighborhoods . Although stacking multiple GATs could in principle enlarge the receptive field and learn non-neighboring interactions , such deep GAT architectures suffer from the oversmoothing problem ( Wang et al. , 2019a ; Liu et al. , 2019b ; Oono & Suzuki , 2020 ) and do not perform well in practice . Furthermore , edge attention weights in the single GAT layer are based solely on representations of the two nodes at the edge endpoints , and do not depend on their graph neighborhood context . In other words , the one-hop attention mechanism in GATs limits their ability to explore the relationship between the broader graph structure and the attention weights . While previous works ( Xu et al. , 2018 ; Klicpera et al. , 2019b ) have shown advantages in performing multi-hop message-passing in a single layer , these approaches are not graph-attention based . Therefore , incorporating multi-hop neighboring context into the attention computation in graph neural networks had not been explored . Here we present Multi-hop Attention Graph Neural Network ( MAGNA ) , an effective and efficient multi-hop self-attention mechanism for graph structured data . MAGNA uses a novel graph attention diffusion layer ( Figure 1 ) , where we first compute attention weights on edges ( represented by solid arrows ) , and then compute self-attention weights ( dotted arrows ) between disconnected pairs of nodes through an attention diffusion process using the attention weights on the edges . Our model has two main advantages : 1 ) MAGNA captures long-range interactions between nodes that are not directly connected but may be multiple hops away . Thus the model enables effective long-range message passing , from important nodes multiple hops away . 2 ) The attention computation in MAGNA is context-dependent . The attention value in GATs ( Veličković et al. , 2018 ) only depends on node representations of the previous layer , and is zero between disconnected pairs of nodes . In contrast , for any pair of nodes within a chosen multi-hop neighborhood , MAGNA computes attention by aggregating the attention scores over all the possible paths ( length≥ 1 ) connecting the two nodes . Theoretically we demonstrate that MAGNA places a Personalized Page Rank ( PPR ) prior on the attention values . We further apply spectral graph analysis to show that MAGNA has the capability of emphasizing on large-scale graph structure and lowering high-frequency noise in graphs . Specifically , MAGNA enlarges the lower Laplacian eigen-values , which correspond to the large-scale structure in the graph , and suppresses the higher Laplacian eigen-values which correspond to more noisy and fine-grained information in the graph . We experiment on standard datasets for semisupervised node classification as well as knowledge graph completion . Experiments show that MAGNA achieves state-of-the-art results : MAGNA achieves up to 5.7 % relative error reduction over previous state-of-the-art on Cora , Citeseer , and Pubmed . MAGNA also obtains better performance on a large-scale Open Graph Benchmark dataset . On knowledge graph completion , MAGNA advances state-of-the-art on WN18RR and FB15k-237 across four metrics , with the largest gain of 7.1 % in the metric of Hit at 1 . Furthermore , we show that MAGNA with just 3 layers and 6 hop wide attention per layer significantly out-performs GAT with 18 layers , even though both architectures have the same receptive field . Moreover , our ablation study reveals the synergistic effect of the essential components of MAGNA , including layer normalization and multi-hop diffused attention . We further observe that compared to GAT , the attention values learned by MAGNA have higher diversity , indicating the ability to better pay attention to important nodes . 2 MULTI-HOP ATTENTION GRAPH NEURAL NETWORK ( MAGNA ) . We first discuss the background and then explain Multi-hop Attention Graph Neural Network ’ s novel multi-hop attention diffusion module and its overall model architecture . 2.1 PRELIMINARIES . Let G = ( V , E ) be a given graph , where V is the set of Nn nodes , E ⊆ V × V is the set of Ne edges connecting M pairs of nodes in V . Each node v ∈ V and each edge e ∈ E are associated with their type mapping functions : φ : V → T and ψ : E → R. Here T and R denote the sets of node types ( labels ) and edge/relation types . Our framework supports learning on heterogeneous graphs with multiple elements inR . A general Graph Neural Network ( GNN ) approach learns an embedding that maps nodes and/or edge types into a continuous vector space . Let X ∈ RNn×dn and R ∈ RNr×dr be the node embedding and edge/relation type embedding , where Nn = |V| , Nr = |R| , dn and dr represent the embedding dimension of node and edge/relation types , each row xi =X [ i : ] represents the embedding of node vi ( 1 ≤ i ≤ Nn ) , and rj = R [ j : ] represents the embedding of relation rj ( 1 ≤ j ≤ Nr ) . MAGNA builds on GNNs , while bringing together the benefits of Graph Attention and Diffusion techniques . The core of MAGNA is Multi-hop Attention Diffusion , a principled way to learn attention between any pair of nodes in a scalable way , taking into account the graph structure and enabling multihop context-dependent attention directly . The key challenge here is how to allow for flexible but scalable context-dependent multi-hop attention , where any node can influence embedding of any other node in a single GNN layer ( even if they are far away in the underlying network ) . Simply learning attention scores over all node pairs is infeasible and would lead to overfitting and poor generalization . 2.2 MULTI-HOP ATTENTION DIFFUSION . We first introduce attention diffusion to compute the multi-hop attention directly , which operates on the MAGNA ’ s attention scores at each layer . The input to the attention diffusion operator is a set of triples ( vi , rk , vj ) , where vi , vj are nodes and rk is the edge type . MAGNA first computes the attention scores on all edges . The attention diffusion module then computes the attention values between pairs of nodes that are not directly connected by an edge , based on the edge attention scores , via a diffusion process . The attention diffusion module can then be used as a component in MAGNA architecture , which we will further elaborate in Section 2.3 . Edge Attention Computation . At each layer l , a vector message is computed for each triple ( vi , rk , vj ) . To compute the representation of vj at layer l + 1 , all messages from triples incident to vj are aggregated into a single message , which is then used to update vl+1j . In the first stage , the attention score s of an edge ( vi , rk , vj ) is computed by the following : s ( l ) i , k , j = LeakyReLU ( v ( l ) a tanh ( W ( l ) h h ( l ) i ‖W ( l ) t h ( l ) j ‖W ( l ) r rk ) ) ( 1 ) whereW ( l ) h , W ( l ) t ∈Rd ( l ) ×d ( l ) , W ( l ) r ∈Rd ( l ) ×dr and v ( l ) a ∈R1×3d ( l ) are the trainable weights shared by l-th layer . h ( l ) i ∈ Rd ( l ) represents the embedding of node i at l-th layer , and h ( 0 ) i = xi . rk is the trainable relation embedding of the k-th relation type rk ( 1 ≤ k ≤ Nr ) , and a‖b denotes concatenation of embedding vectors a and b . For graphs with no relation type , we treat as a degenerate categorical distribution with 1 category 1 . Applying Eq . 1 on each edge of the graph G , we obtain an attention score matrix S ( l ) : S ( l ) i , j = { s ( l ) i , k , j , if ( vi , rk , vj ) appears in G −∞ , otherwise ( 2 ) Subsequently we obtain the attention matrix A ( l ) by performing row-wised softmax over the score matrix S ( l ) : A ( l ) = softmax ( S ( l ) ) . A ( l ) ij denotes the attention value at layer l when aggregating message from node j to node i . Attention Diffusion for Multi-hop Neighbors . In the second stage , we further enable attention between nodes that are not directly connected in the network . We achieve this via the following attention diffusion procedure . The procedure computes the attention scores of multi-hop neighbors via graph diffusion based on the powers of the 1-hop attention matrixA : A = ∞∑ i=0 θiA i where ∞∑ i=0 θi = 1 and θi > 0 ( 3 ) where θi is the attention decay factor and θi > θi+1 . The powers of attention matrix , Ai , give us the number of relation paths from node h to node t of length up to i , increasing the receptive field of the attention ( Figure 1 ) . Importantly , the mechanism allows the attention between two nodes to not only depend on their previous layer representations , but also taking into account of the paths between the 1In this case , we can view that there is only one “ pseudo ” relation type ( category ) , i.e. , Nr = 1 nodes , effectively creating attention shortcuts between nodes that are not directly connected ( Figure 1 ) . Attention through each path is also weighted differently , depending on θ and the path length . In our implementation we utilize the geometric distribution θi = α ( 1 − α ) i , where α ∈ ( 0 , 1 ] . The choice is based on the inductive bias that nodes further away should be weighted less in message aggregation , and nodes with different relation path lengths to the target node are sequentially weighted in an independent manner . In addition , notice that if we define θ0 = α ∈ ( 0 , 1 ] , A0 = I , then Eq . 3 gives the Personalized Page Rank ( PPR ) procedure on the graph with the attention matrix A and teleport probability α . Hence the diffused attention weights , Aij , can be seen as the influence of node j to node i . We further elaborate the significance of this observation in Section 4.3 . We can also viewAij as the attention value of node j to i since ∑Nn j=1Aij = 1.2 We then define the graph attention diffusion based feature aggregation as AttDiffusion ( G , H ( l ) , Θ ) = AH ( l ) , ( 4 ) where Θ is the set of parameters for computing attention . Thanks to the diffusion process defined in Eq . 3 , MAGNA uses the same number of parameters as if we were only computing attention between nodes connected via edges . This ensures runtime efficiency as well as good model generalization . Approximate Computation for Attention Diffusion . For large graphs computing the exact attention diffusion matrix A using Eq . 3 may be prohibitively expensive , due to computing the powers of the attention matrix ( Klicpera et al. , 2019a ) . To resolve this bottleneck , we proceed as follows : Let H ( l ) be the input entity embedding of the l-th layer ( H ( 0 ) = X ) and θi = α ( 1 − α ) i . Since MAGNA only requires aggregation via AH ( l ) , we can approximate AH ( l ) by defining a sequence Z ( K ) which converges to the true value of AH ( l ) ( Proposition 1 ) as K →∞ : Z ( 0 ) =H ( l ) , Z ( k+1 ) = ( 1− α ) AZ ( k ) + αZ ( 0 ) , where 0 ≤ k < K ( 5 ) Proposition 1. limK→∞ Z ( K ) = AH ( l ) In the Appendix we give the proof which relies on the expansion of Eq . 5 . Using the above approximation , the complexity of attention computation with diffusion is still O ( |E| ) , with a constant factor corresponding to the number of hops K. In practice , we find that choosing the values of K such that 3 ≤ K ≤ 10 results in good model performance . Many realworld graphs exhibit small-world property , in which case even a smaller value of K is sufficient . For graphs with larger diameter , we choose larger K , and lower the value of α .

Summary:      Conventional Graph Neural Networks (GNNs) learn node representations that encode information from multiple hops away by iteratively aggregating information through their immediate neighbors. Self-Attention modules have been adopted to GNNs to selectively aggregate information coming through the immediate neighbors at different propagation stages. However, current self-attention mechanisms are limited to only attend over the nodes' immediate neighbors and not directly over their neighbors that are multiple hops away. Here in this work, the authors intend to address this issue and propose a means to obtain attention scores over indirectly connected neighbors.