How Attentive are Graph Attention Networks? 

1 INTRODUCTION . Graph neural networks ( GNNs ; Gori et al. , 2005 ; Scarselli et al. , 2008 ) have seen increasing popularity over the past few years ( Duvenaud et al. , 2015 ; Atwood and Towsley , 2016 ; Bronstein et al. , 2017 ; Monti et al. , 2017 ) . GNNs provide a general and efficient framework to learn from graph-structured data . Thus , GNNs are easily applicable in domains where the data can be represented as a set of nodes and the prediction depends on the relationships ( edges ) between the nodes . Such domains include molecules , social networks , product recommendation , computer programs and more . In a GNN , each node iteratively updates its state by interacting with its neighbors . GNN variants ( Wu et al. , 2019 ; Xu et al. , 2019 ; Li et al. , 2016 ) mostly differ in how each node aggregates and combines the representations of its neighbors with its own . Veličković et al . ( 2018 ) pioneered the use of attention-based neighborhood aggregation , in one of the most common GNN variants – Graph Attention Network ( GAT ) . In GAT , every node updates its representation by attending to its neighbors using its own representation as the query . This generalizes the standard averaging or max-pooling of neighbors ( Kipf and Welling , 2017 ; Hamilton et al. , 2017 ) , by allowing every node to compute a weighted average of its neighbors , and ( softly ) select its most relevant neighbors . The work of 1An annotated implementation of GATv2 is available at https : //nn.labml.ai/graphs/gatv2/ 2from torch_geometric.nn.conv.gatv2_conv import GATv2Conv 3from dgl.nn.pytorch import GATv2Conv 4from tensorflow_gnn.graph.keras.layers.gat_v2 import GATv2Convolution k0 k1 k2 k3 k4 k5 k6 k7 k8 k9 q0 q1 q2 q3 q4 q5 q6 q7 q8 q9 0.08 0.10 0.10 0.07 0.08 0.08 0.11 0.09 0.20 0.08 0.05 0.10 0.10 0.04 0.04 0.04 0.13 0.06 0.38 0.04 0.05 0.10 0.10 0.04 0.05 0.05 0.13 0.06 0.38 0.05 0.08 0.10 0.10 0.07 0.08 0.08 0.10 0.09 0.24 0.08 0.08 0.09 0.09 0.07 0.07 0.07 0.10 0.08 0.27 0.07 0.09 0.11 0.11 0.08 0.09 0.08 0.11 0.10 0.16 0.09 0.04 0.10 0.11 0.03 0.04 0.04 0.14 0.06 0.40 0.04 0.07 0.09 0.09 0.06 0.07 0.07 0.10 0.08 0.29 0.07 0.04 0.11 0.11 0.02 0.04 0.03 0.14 0.07 0.41 0.04 0.07 0.09 0.09 0.06 0.07 0.07 0.11 0.08 0.30 0.07 k0 k1 k2 k3 k4 k5 k6 k7 k8 k9 0.1 0.2 0.3 0.4 q0 q1 q2 q3 q4 q5 q6 q7 q8 q9 ( a ) Attention in standard GAT ( Veličković et al . ( 2018 ) ) k0 k1 k2 k3 k4 k5 k6 k7 k8 k9 q0 q1 q2 q3 q4 q5 q6 q7 q8 q9 0.95 0.00 0.00 0.01 0.01 0.00 0.00 0.02 0.01 0.00 0.01 0.92 0.01 0.01 0.01 0.00 0.01 0.01 0.00 0.02 0.00 0.00 0.95 0.00 0.00 0.01 0.02 0.01 0.00 0.00 0.01 0.01 0.00 0.94 0.00 0.01 0.00 0.00 0.02 0.01 0.00 0.00 0.00 0.00 0.96 0.00 0.00 0.01 0.01 0.00 0.00 0.01 0.01 0.01 0.01 0.89 0.01 0.01 0.04 0.02 0.00 0.01 0.04 0.00 0.01 0.01 0.86 0.02 0.01 0.03 0.04 0.02 0.01 0.01 0.03 0.01 0.00 0.87 0.00 0.01 0.01 0.00 0.01 0.01 0.01 0.01 0.01 0.00 0.94 0.00 0.01 0.02 0.01 0.01 0.01 0.01 0.01 0.00 0.00 0.93 k0 k1 k2 k3 k4 k5 k6 k7 k8 k9 0.0 0.2 0.4 0.6 0.8 1.0 q0 q1 q2 q3 q4 q5 q6 q7 q8 q9 ( b ) Attention in GATv2 , our fixed version of GAT Figure 1 : In a complete bipartite graph of “ query nodes ” { q0 , ... , q9 } and “ key nodes ” { k0 , ... , k9 } : standard GAT ( Figure 1a ) computes static attention – the ranking of attention coefficients is global for all nodes in the graph , and is unconditioned on the query node . For example , all queries ( q0 to q9 ) attend mostly to the 8th key ( k8 ) . In contrast , GATv2 ( Figure 1b ) can actually compute dynamic attention , where every query has a different ranking of attention coefficients of the keys . Veličković et al . also generalizes the Transformer ’ s ( Vaswani et al. , 2017 ) self-attention mechanism , from sequences to graphs ( Joshi , 2020 ) . Nowadays , GAT is one of the most popular GNN architectures ( Bronstein et al. , 2021 ) and is considered as the state-of-the-art neural architecture for learning with graphs ( Wang et al. , 2019a ) . Nevertheless , in this paper we show that GAT does not actually compute the expressive , well known , type of attention ( Bahdanau et al. , 2014 ) , which we call dynamic attention . Instead , we show that GAT computes only a restricted “ static ” form of attention : for any query node , the attention function is monotonic with respect to the neighbor ( key ) scores . That is , the ranking ( the argsort ) of attention coefficients is identical for all nodes in the graph , and is unconditioned on the query node . This fact severely hurts the expressiveness of GAT , and is demonstrated in Figure 1a . Supposedly , the conceptual idea of attention as the form of interaction between GNN nodes is orthogonal to the specific choice of attention function . However , Veličković et al. ’ s original design of GAT has spread to a variety of domains ( Yang et al. , 2020 ; Wang et al. , 2019c ; Huang and Carley , 2019 ; Ma et al. , 2020 ; Kosaraju et al. , 2019 ; Nathani et al. , 2019 ; Wu et al. , 2020 ; Zhang et al. , 2020 ) and has become the default implementation of “ graph attention network ” in all popular GNN libraries such as PyTorch Geometric ( Fey and Lenssen , 2019 ) , DGL ( Wang et al. , 2019b ) , and others ( Dwivedi et al. , 2020 ; Gordić , 2020 ; Brockschmidt , 2020 ) . Many other works employed GNNs with attention mechanisms other than the standard GAT ’ s ( Yun et al . ( 2019 ) , see also Appendix A ) . However , none of these works identified the monotonicity of GAT ’ s attention mechanism , the theoretical differences between attention types , nor empirically compared their performance . To overcome the limitation we identified in GAT , we introduce a simple fix to its attention function by only modifying the order of internal operations . The result is GATv2 – a graph attention variant that has a universal approximator attention function , and is thus strictly more expressive than GAT . The effect of fixing the attention function in GATv2 is demonstrated in Figure 1b . In summary , our main contribution is identifying that one of the most popular GNN types , the graph attention network , does not compute dynamic attention , the kind of attention that it seems to compute . We introduce formal definitions for analyzing the expressive power of graph attention mechanisms ( Definitions 3.1 and 3.2 ) , and derive our claims theoretically ( Theorem 1 ) from the equations of Veličković et al . ( 2018 ) . Empirically , we use a synthetic problem to show that standard GAT can not express problems that require dynamic attention ( Section 4.1 ) . We introduce a simple fix by switching the order of internal operations in GAT , and propose GATv2 , which does compute dynamic attention ( Theorem 2 ) . We further conduct a thorough empirical comparison of GAT and GATv2 and find that GATv2 outperforms GAT across 12 benchmarks of node- , link- , and graph-prediction . For example , GATv2 outperforms extensively tuned GNNs by over 1.4 % in the difficult “ UnseenProj Test ” set of the VarMisuse task ( Allamanis et al. , 2018 ) , without any hyperparameter tuning ; and GATv2 improves over an extensively-tuned GAT by 11.5 % in 13 prediction objectives in QM9 . In node-prediction benchmarks from OGB ( Hu et al. , 2020 ) , not only that GATv2 outperforms GAT with respect to accuracy – we find that dynamic attention provided a much better robustness to noise . 2 PRELIMINARIES . A directed graph G = ( V , E ) contains nodes V = { 1 , ... , n } and edges E ⊆ V × V , where ( j , i ) ∈ E denotes an edge from a node j to a node i . We assume that every node i ∈ V has an initial representation h ( 0 ) i ∈ Rd0 . An undirected graph can be represented with bidirectional edges . 2.1 GRAPH NEURAL NETWORKS . A graph neural network ( GNN ) layer updates every node representation by aggregating its neighbors ’ representations . A layer ’ s input is a set of node representations { hi ∈ Rd | i ∈ V } and the set of edges E . A layer outputs a new set of node representations { h′i ∈ Rd ′ | i ∈ V } , where the same parametric function is applied to every node given its neighbors Ni = { j ∈ V | ( j , i ) ∈ E } : h′i = fθ ( hi , AGGREGATE ( { hj | j ∈ Ni } ) ) ( 1 ) The design of f and AGGREGATE is what mostly distinguishes one type of GNN from the other . For example , a common variant of GraphSAGE ( Hamilton et al. , 2017 ) performs an element-wise mean as AGGREGATE , followed by concatenation with hi , a linear layer and a ReLU as f . 2.2 GRAPH ATTENTION NETWORKS . GraphSAGE and many other popular GNN architectures ( Xu et al. , 2019 ; Duvenaud et al. , 2015 ) weigh all neighbors j ∈ Ni with equal importance ( e.g. , mean or max-pooling as AGGREGATE ) . To address this limitation , GAT ( Veličković et al. , 2018 ) instantiates Equation ( 1 ) by computing a learned weighted average of the representations ofNi . A scoring function e : Rd×Rd→R computes a score for every edge ( j , i ) , which indicates the importance of the features of the neighbor j to the node i : e ( hi , hj ) = LeakyReLU ( a > · [ Whi‖Whj ] ) ( 2 ) where a ∈ R2d′ , W ∈ Rd′×d are learned , and ‖ denotes vector concatenation . These attention scores are normalized across all neighbors j ∈ Ni using softmax , and the attention function is defined as : αij = softmaxj ( e ( hi , hj ) ) = exp ( e ( hi , hj ) ) ∑ j′∈Ni exp ( e ( hi , hj′ ) ) ( 3 ) Then , GAT computes a weighted average of the transformed features of the neighbor nodes ( followed by a nonlinearity σ ) as the new representation of i , using the normalized attention coefficients : h′i = σ ( ∑ j∈Ni αij ·Whj ) ( 4 ) From now on , we will refer to Equations ( 2 ) to ( 4 ) as the definition of GAT . 3 THE EXPRESSIVE POWER OF GRAPH ATTENTION MECHANISMS . In this section , we explain why attention is limited when it is not dynamic ( Section 3.1 ) . We then show that GAT is severely constrained , because it can only compute static attention ( Section 3.2 ) . Next , we show how GAT can be fixed ( Section 3.3 ) , by simply modifying the order of operations . We refer to a neural architecture ( e.g. , the scoring or the attention function of GAT ) as a family of functions , parameterized by the learned parameters . An element in the family is a concrete function with specific trained weights . In the following , we use [ n ] to denote the set [ n ] = { 1 , 2 , ... , n } ⊂ N .

Summary: This paper analyzes the limitation of GAT by pointing out that GAT computes a limited kind of attention: static attention. This paper then introduces a simple fix by modifying the order of operations and proposes GATv2: a dynamic attention variant. Experiments show that GATv2 can outperform GAT.