Multi-hop Attention Graph Neural Network

1 INTRODUCTION . The introduction of the self-attention mechanism ( Bahdanau et al. , 2015 ; Vaswani et al. , 2017 ) , has pushed the state-of-the-art in many domains including graph presentation learning ( Radford et al. , 2019 ; Devlin et al. , 2019 ; Liu et al. , 2019a ; Lan et al. , 2019 ) . Graph Attention Network ( GAT ) ( Veličković et al. , 2018 ) and related models ( Li et al. , 2018 ; Wang et al. , 2019a ; Liu et al. , 2019b ; Oono & Suzuki , 2020 ) developed attention mechanism for Graph Neural Networks ( GNNs ) , which compute attention scores between nodes connected by an edge , allowing the model to attend to messages of node ’ s direct neighbors according to their attention scores . However , such attention computation on pairs of nodes connected by edges implies that a node can only attend to its immediate neighbors to compute its ( next layer ) representation . This implies that receptive field of a single GNN layer is restricted to one-hop network neighborhoods . Although stacking multiple GATs could in principle enlarge the receptive field and learn non-neighboring interactions , such deep GAT architectures suffer from the oversmoothing problem ( Wang et al. , 2019a ; Liu et al. , 2019b ; Oono & Suzuki , 2020 ) and do not perform well in practice . Furthermore , edge attention weights in the single GAT layer are based solely on representations of the two nodes at the edge endpoints , and do not depend on their graph neighborhood context . In other words , the one-hop attention mechanism in GATs limits their ability to explore the relationship between the broader graph structure and the attention weights . While previous works ( Xu et al. , 2018 ; Klicpera et al. , 2019b ) have shown advantages in performing multi-hop message-passing in a single layer , these approaches are not graph-attention based . Therefore , incorporating multi-hop neighboring context into the attention computation in graph neural networks had not been explored . Here we present Multi-hop Attention Graph Neural Network ( MAGNA ) , an effective and efficient multi-hop self-attention mechanism for graph structured data . MAGNA uses a novel graph attention diffusion layer ( Figure 1 ) , where we first compute attention weights on edges ( represented by solid arrows ) , and then compute self-attention weights ( dotted arrows ) between disconnected pairs of nodes through an attention diffusion process using the attention weights on the edges . Our model has two main advantages : 1 ) MAGNA captures long-range interactions between nodes that are not directly connected but may be multiple hops away . Thus the model enables effective long-range message passing , from important nodes multiple hops away . 2 ) The attention computation in MAGNA is context-dependent . The attention value in GATs ( Veličković et al. , 2018 ) only depends on node representations of the previous layer , and is zero between disconnected pairs of nodes . In contrast , for any pair of nodes within a chosen multi-hop neighborhood , MAGNA computes attention by aggregating the attention scores over all the possible paths ( length≥ 1 ) connecting the two nodes . Theoretically we demonstrate that MAGNA places a Personalized Page Rank ( PPR ) prior on the attention values . We further apply spectral graph analysis to show that MAGNA has the capability of emphasizing on large-scale graph structure and lowering high-frequency noise in graphs . Specifically , MAGNA enlarges the lower Laplacian eigen-values , which correspond to the large-scale structure in the graph , and suppresses the higher Laplacian eigen-values which correspond to more noisy and fine-grained information in the graph . We experiment on standard datasets for semisupervised node classification as well as knowledge graph completion . Experiments show that MAGNA achieves state-of-the-art results : MAGNA achieves up to 5.7 % relative error reduction over previous state-of-the-art on Cora , Citeseer , and Pubmed . MAGNA also obtains better performance on a large-scale Open Graph Benchmark dataset . On knowledge graph completion , MAGNA advances state-of-the-art on WN18RR and FB15k-237 across four metrics , with the largest gain of 7.1 % in the metric of Hit at 1 . Furthermore , we show that MAGNA with just 3 layers and 6 hop wide attention per layer significantly out-performs GAT with 18 layers , even though both architectures have the same receptive field . Moreover , our ablation study reveals the synergistic effect of the essential components of MAGNA , including layer normalization and multi-hop diffused attention . We further observe that compared to GAT , the attention values learned by MAGNA have higher diversity , indicating the ability to better pay attention to important nodes . 2 MULTI-HOP ATTENTION GRAPH NEURAL NETWORK ( MAGNA ) . We first discuss the background and then explain Multi-hop Attention Graph Neural Network ’ s novel multi-hop attention diffusion module and its overall model architecture . 2.1 PRELIMINARIES . Let G = ( V , E ) be a given graph , where V is the set of Nn nodes , E ⊆ V × V is the set of Ne edges connecting M pairs of nodes in V . Each node v ∈ V and each edge e ∈ E are associated with their type mapping functions : φ : V → T and ψ : E → R. Here T and R denote the sets of node types ( labels ) and edge/relation types . Our framework supports learning on heterogeneous graphs with multiple elements inR . A general Graph Neural Network ( GNN ) approach learns an embedding that maps nodes and/or edge types into a continuous vector space . Let X ∈ RNn×dn and R ∈ RNr×dr be the node embedding and edge/relation type embedding , where Nn = |V| , Nr = |R| , dn and dr represent the embedding dimension of node and edge/relation types , each row xi =X [ i : ] represents the embedding of node vi ( 1 ≤ i ≤ Nn ) , and rj = R [ j : ] represents the embedding of relation rj ( 1 ≤ j ≤ Nr ) . MAGNA builds on GNNs , while bringing together the benefits of Graph Attention and Diffusion techniques . The core of MAGNA is Multi-hop Attention Diffusion , a principled way to learn attention between any pair of nodes in a scalable way , taking into account the graph structure and enabling multihop context-dependent attention directly . The key challenge here is how to allow for flexible but scalable context-dependent multi-hop attention , where any node can influence embedding of any other node in a single GNN layer ( even if they are far away in the underlying network ) . Simply learning attention scores over all node pairs is infeasible and would lead to overfitting and poor generalization . 2.2 MULTI-HOP ATTENTION DIFFUSION . We first introduce attention diffusion to compute the multi-hop attention directly , which operates on the MAGNA ’ s attention scores at each layer . The input to the attention diffusion operator is a set of triples ( vi , rk , vj ) , where vi , vj are nodes and rk is the edge type . MAGNA first computes the attention scores on all edges . The attention diffusion module then computes the attention values between pairs of nodes that are not directly connected by an edge , based on the edge attention scores , via a diffusion process . The attention diffusion module can then be used as a component in MAGNA architecture , which we will further elaborate in Section 2.3 . Edge Attention Computation . At each layer l , a vector message is computed for each triple ( vi , rk , vj ) . To compute the representation of vj at layer l + 1 , all messages from triples incident to vj are aggregated into a single message , which is then used to update vl+1j . In the first stage , the attention score s of an edge ( vi , rk , vj ) is computed by the following : s ( l ) i , k , j = LeakyReLU ( v ( l ) a tanh ( W ( l ) h h ( l ) i ‖W ( l ) t h ( l ) j ‖W ( l ) r rk ) ) ( 1 ) whereW ( l ) h , W ( l ) t ∈Rd ( l ) ×d ( l ) , W ( l ) r ∈Rd ( l ) ×dr and v ( l ) a ∈R1×3d ( l ) are the trainable weights shared by l-th layer . h ( l ) i ∈ Rd ( l ) represents the embedding of node i at l-th layer , and h ( 0 ) i = xi . rk is the trainable relation embedding of the k-th relation type rk ( 1 ≤ k ≤ Nr ) , and a‖b denotes concatenation of embedding vectors a and b . For graphs with no relation type , we treat as a degenerate categorical distribution with 1 category 1 . Applying Eq . 1 on each edge of the graph G , we obtain an attention score matrix S ( l ) : S ( l ) i , j = { s ( l ) i , k , j , if ( vi , rk , vj ) appears in G −∞ , otherwise ( 2 ) Subsequently we obtain the attention matrix A ( l ) by performing row-wised softmax over the score matrix S ( l ) : A ( l ) = softmax ( S ( l ) ) . A ( l ) ij denotes the attention value at layer l when aggregating message from node j to node i . Attention Diffusion for Multi-hop Neighbors . In the second stage , we further enable attention between nodes that are not directly connected in the network . We achieve this via the following attention diffusion procedure . The procedure computes the attention scores of multi-hop neighbors via graph diffusion based on the powers of the 1-hop attention matrixA : A = ∞∑ i=0 θiA i where ∞∑ i=0 θi = 1 and θi > 0 ( 3 ) where θi is the attention decay factor and θi > θi+1 . The powers of attention matrix , Ai , give us the number of relation paths from node h to node t of length up to i , increasing the receptive field of the attention ( Figure 1 ) . Importantly , the mechanism allows the attention between two nodes to not only depend on their previous layer representations , but also taking into account of the paths between the 1In this case , we can view that there is only one “ pseudo ” relation type ( category ) , i.e. , Nr = 1 nodes , effectively creating attention shortcuts between nodes that are not directly connected ( Figure 1 ) . Attention through each path is also weighted differently , depending on θ and the path length . In our implementation we utilize the geometric distribution θi = α ( 1 − α ) i , where α ∈ ( 0 , 1 ] . The choice is based on the inductive bias that nodes further away should be weighted less in message aggregation , and nodes with different relation path lengths to the target node are sequentially weighted in an independent manner . In addition , notice that if we define θ0 = α ∈ ( 0 , 1 ] , A0 = I , then Eq . 3 gives the Personalized Page Rank ( PPR ) procedure on the graph with the attention matrix A and teleport probability α . Hence the diffused attention weights , Aij , can be seen as the influence of node j to node i . We further elaborate the significance of this observation in Section 4.3 . We can also viewAij as the attention value of node j to i since ∑Nn j=1Aij = 1.2 We then define the graph attention diffusion based feature aggregation as AttDiffusion ( G , H ( l ) , Θ ) = AH ( l ) , ( 4 ) where Θ is the set of parameters for computing attention . Thanks to the diffusion process defined in Eq . 3 , MAGNA uses the same number of parameters as if we were only computing attention between nodes connected via edges . This ensures runtime efficiency as well as good model generalization . Approximate Computation for Attention Diffusion . For large graphs computing the exact attention diffusion matrix A using Eq . 3 may be prohibitively expensive , due to computing the powers of the attention matrix ( Klicpera et al. , 2019a ) . To resolve this bottleneck , we proceed as follows : Let H ( l ) be the input entity embedding of the l-th layer ( H ( 0 ) = X ) and θi = α ( 1 − α ) i . Since MAGNA only requires aggregation via AH ( l ) , we can approximate AH ( l ) by defining a sequence Z ( K ) which converges to the true value of AH ( l ) ( Proposition 1 ) as K →∞ : Z ( 0 ) =H ( l ) , Z ( k+1 ) = ( 1− α ) AZ ( k ) + αZ ( 0 ) , where 0 ≤ k < K ( 5 ) Proposition 1. limK→∞ Z ( K ) = AH ( l ) In the Appendix we give the proof which relies on the expansion of Eq . 5 . Using the above approximation , the complexity of attention computation with diffusion is still O ( |E| ) , with a constant factor corresponding to the number of hops K. In practice , we find that choosing the values of K such that 3 ≤ K ≤ 10 results in good model performance . Many realworld graphs exhibit small-world property , in which case even a smaller value of K is sufficient . For graphs with larger diameter , we choose larger K , and lower the value of α .

Summary: This paper proposes MAGNA, a multi-hop self-attention mechanism for attention based graph neural networks. The proposed method increases the receptive field at each layer, requiring less layers to achieve a large receptive field. Also, with the proposed method the attention coefficient between two nodes is not just a function of the two nodes but also of their neighbourhood. The proposed MAGNA method is an extension of GAT networks that introduces a diffusion step on the computed attention coefficients, following a similar approach (Diffusion-GCNs) that has been used for GCNs.