Workshop on Representation Learning on Graphs and Manifolds U NDERSTANDING ATTENTION IN GRAPH NEURAL NETWORKS

We aim to better understand attention over nodes in graph neural networks and identify factors influencing its effectiveness. Motivated by insights from the work on Graph Isomorphism Networks (Xu et al., 2019), we design simple graph reasoning tasks that allow us to study attention in a controlled environment. We find that under typical conditions the effect of attention is negligible or even harmful, but under certain conditions it provides an exceptional gain in performance of more than 40% in some of our classification tasks. However, we have yet to satisfy these conditions in practice. 1 ATTENTION MEETS POOLING IN GRAPH NEURAL NETWORKS The practical importance of attention in deep learning is well-established and there are many arguments in its favor (Vaswani et al., 2017), including interpretability (Park et al., 2016; Deac et al., 2018). In graph neural networks (GNNs), attention can be defined over edges (Velickovic et al., 2018; Zhang et al., 2018) or over nodes (Lee et al., 2018a). In this work, we focus on the latter, because, despite being equally important in certain tasks, it is not thoroughly studied in previous work (Lee et al., 2018b). To start our description, we first establish a connection between attention and pooling methods. In convolutional neural networks (CNNs), pooling methods are generally based on uniformly dividing the regular grid (such as one-dimensional temporal grid in audio) into local regions and taking a single value from that region (average, weighted average, max, stochastic, etc.), while attention in CNNs is typically a separate mechanism that weights input X ∈ RN×C : Z = α X, (1) where Zi = αiXi output for unit (node in a graph) i, ∑N i αi = 1, element-wise multiplication, N the number of units in the input (i.e. number of nodes in a graph), C its dimensionality. In GNNs, pooling methods generally follow the same pattern as in CNNs, but the pooling regions (sets of nodes) are found based on clustering (Defferrard et al., 2016; Ying et al., 2018), since there is no grid that can be uniformly divided into regions in the same way across all examples (graphs) in the dataset. Recently, top-k pooling (Gao & Ji, 2018) was proposed, diverging from other methods: instead of clustering “similar” nodes, it propagates only part of the input and this part is not uniformly sampled from the input. Top-k pooling can thus select some local part of the input graph, completely ignoring the rest. For this reason at first glance it does not appear to be logical. However, we can notice that pooled feature maps in Gao & Ji (2018, Eq. 2) are computed in the same way as attention outputs Z in Eq. 1 above, if we rewrite their Eq. 2 in the following way: Zi = { αiXi, ∀i ∈ P ∅, otherwise (2) where P is a set of indices of pooled nodes, |P | ≤ N , and ∅ denotes the unit is absent in the output. The only difference between Eq. 2 and Eq. 1 is that Z ∈ R|P |×C , i.e. the number of units in the output is smaller or, formally, there exists a ratio r = |P |/N ≤ 1 of preserved nodes. Accordingly, we integrate attention and pooling into a single computational block of a GNN. In contrast, in CNNs,