Characterize and Transfer Attention in Graph Neural Networks

1 INTRODUCTION . The modeling of graphs has become an active research topic in deep learning ( Bronstein et al. , 2017 ) . Dozens of neural network models have been developed for exploiting the structural information of graphs ( Scarselli et al. , 2009 ; Bruna et al. , 2014 ; Henaff et al. , 2015 ; Duvenaud et al. , 2015 ; Niepert et al. , 2016 ; Defferrard et al. , 2016 ) , now collectively referred to as graph neural networks ( GNNs ) . Built upon the success of attention in NLP ( Vaswani et al. , 2017 ) , Veličković et al . ( 2018 ) proposed the graph attention networks ( GATs ) to integrate multi-head self-attention into node feature update for adaptive weighting , with several extensions ( Thekumparampil et al. , 2018 ; Zhang et al. , 2018 ; Monti et al. , 2018 ; Svoboda et al. , 2019 ; Trivedi et al. , 2019 ) . While the use of attention in GNNs is an attractive direction , several works also report that attention contributes little to the performance of GNNs ( Zhang et al. , 2018 ; Shchur et al. , 2019 ) . Considering the high computational cost of attention , the question is then that does attention help and , if so , when and how ? In this paper , we take a first step towards the question . We first identify the key questions for understanding attention and propose an analytical paradigm . With extensive experiments , our findings suggest that , although attention is motivated by inductive learning , its functionality depends highly on the characteristics of the datasets . The attention distributions across heads and layers are near uniform for all citation networks ( Cora , Citeseer and Pubmed ) while they get more concentrated over layers on the protein-protein interaction networks ( PPI ) and molecular graphs , with significant diversity among heads . That the attention distribution is a telltale sign of the nature of graph class is further verified with a meta graph classification experiment . With attention features as inputs , citation networks are indistinguishable whereas PPI and molecule graphs are . Inspired by these findings , we hypothesize that attention carry semantic meanings when they are non-uniform and can be helpful for transfer learning . This has been the case in the NLP community ( Radford et al. , 2019 ) , and is motivating many research efforts on understanding multi-head attention ( Jain & Wallace , 2019 ; Clark et al. , 2019 ; Voita et al. , 2019 ) . We attempt the idea of attention based sparsification – sparsifying a graph by retaining edges where attention are higher , with the intuition being that the resulting graph preserves enough information . We find that not only such attention-based sparsification is transferable ( meaning , it can work on unseen graphs ) , it also affords us to train a cheaper model without using attention to fit the downstream task . Finally , we discuss several possible fruitful directions for further exploration , including theory , interpretability , and unsupervised learning . 2 RELATED WORK . Visualize and understand attention Several works attempted to visualize the learned attention by coloring edges or nodes based on the attention magnitudes ( Veličković et al. , 2018 ; Qiu et al. , 2018 ) . Thekumparampil et al . ( 2018 ) studied the averaged attention values between nodes with different or the same class labels . Shanthamallu et al . ( 2018 ) studied the attention GAT learned on two citation networks Cora and Citeseer with interquartile range metric and showed that they are near uniform . Knyazev et al . ( 2019 ) investigated the effectiveness of attentional pooling and find that attention is only effective when it is close to optimal . Our work differs in that we propose a general paradigm for analyzing graph attention , including how to characterize the overall attention statistics and how to measure the layer-wise and head-wise differences of the learned attention . Transfer attention In the computer vision community , transferring the attention maps using a teacher-student network to improve the downstream tasks is a well-studied technique ( Zagoruyko & Komodakis , 2017 ; Li et al. , 2019 ) . Our approach for transferring attention is different from these works in that we use the trained graph attention network as a graph sparsifier instead of a teaching signal . Yang et al . ( 2018 ) proposed to transfer the relational structure within the data , which is represented as a set of attention weights , to boost the performance of other tasks . Our transferring strategy is different from their work because we reduce the sparsity of the affinity matrix by removing the entries with smaller attention weights . Thus , we can train a cheap graph sparsifier to accelerate the training and testing speed . 3 BACKGROUND . 3.1 GRAPH NEURAL NETWORKS . Let G be an undirected graph with node set V and edge set E , where each node i ∈ V has a feature h0i ∈ Rn0 . In a wide class of GNNs ( Kipf & Welling , 2017 ; Hamilton et al. , 2017 ; Veličković et al. , 2018 ) , the basic feature update function for node i ∈ V at the l + 1-th GNN layer takes the form of hl+1i = σ ∑ j∈N ( i ) αl+1i , j W l+1hlj , where σ is an activation function , N ( i ) is a set containing i and its neighbors , αl+1i , j ∈ R is the attention weight of edge ( j , i ) in updating the feature of node i , Wl+1 ∈ Rnl+1×nl is the projection matrix , and hli , h l+1 i are corresponding node features after the l-th and the l + 1-th layer . With a sparse implementation , it has a time complexity of O ( |V|nl+1nl + |E|nl+1 ) . Graph Convolutional Network ( GCN ) ( Kipf & Welling , 2017 ) and the mean variant of GraphSAGE ( Hamilton et al. , 2017 ) use static attention 1√ |N ( i ) | 1√ |N ( j ) | and 1|N ( i ) | , which we separately refer to as GCN and uniform attention . GAT ( Veličković et al. , 2018 ) uses a parameterized subnetwork to output the attention weights αi , js . Rather than using a single attention head as in Eqn . equation 3.1 , GAT aggregates the outputs of multiple heads : αl+1 , ki , j = exp ( score ( hli , h l j ) ) ∑ j′∈N ( i ) exp ( score ( hli , h l j′ ) ) , hl+1 , ki = σ ∑ j∈N ( i ) αl+1 , ki , j W l+1 , khlj , hl+1i = σ ( Aggregatel+1 ( hl+1,1i , · · · , h l+1 , Kl+1 i ) ) , where k is the index of the attention head and Kl+1 is the number of attention heads in the l + 1-th layer . Aggregatel+1 aggregates all head results in the l + 1-th layer and we follow the approach of Veličković et al . ( 2018 ) to use concatenation for intermediate layers and average for the final layer . Attention variants As in Luong et al . ( 2015 ) , there are multiple ways to calculate the attention scores . In this paper , we focus on the following three types of attention , namely concat , dot product , Table 1 : Dataset task and learning setting Cora Citeseer Pubmed PPI CEP HIV Task Node Classification 3 3 3 3Graph Prediction 3 3 Setting Transductive Learning 3 3 3Inductive Learning 3 3 3 and general : LReLU ( aT [ Whi||Whj ] ) ( concat ) , ( Whi ) T Whj ( dot product ) , ( Whi ) T BWhj ( general ) GATs uses the concat attention . Zhang et al . ( 2018 ) and Ryu et al . ( 2018 ) separately explores dot product and general attention in GNNs . Also , LReLU ( · ) means the leaky ReLU activation . Graph-level prediction Based on the node representations generated by GNNs , we can also compute a graph representation ( Li et al. , 2018 ) for graph-level prediction problems like graph classification and regression : hG = ∑ v∈V Sigmoid ( g ( hLv ) ) ReLU ( f ( hLv ) ) , where L is the number of GNN layers , hLv is the representation of node v output by the last layer of GNN , g ( · ) calculates the impact of node v on the graph representation , and f ( · ) is a linear projection . 3.2 TASKS AND DATASETS . We consider the tasks of node classification and graph-level prediction . For modeling , we treat all graphs as undirected with untyped nodes and edges . Self loops are added to preserve information from previous node features . As in Veličković et al . ( 2018 ) , we consider four datasets – citation networks Cora , Citeseer ( Sen et al. , 2008 ) , Pubmed ( Namata et al. , 2012 ) and PPI ( Zitnik & Leskovec , 2017 ) . Additionally , we include two more datasets of molecular graphs for graph-level prediction . The Harvard Clean Energy Project ( CEP ) ( Hachmann et al. , 2011 ) estimates the photovoltaic efficiency of organic molecules . We use a subset of it pre-processed by Duvenaud et al . ( 2015 ) ; Ryu et al . ( 2018 ) . The HIV dataset was initially introduced by the Drug Therapeutics Program ( DTP ) AIDS Antiviral Screen HIV for testing the ability of compounds to inhibit HIV replication . It was later included in the MoleculeNet benchmark ( Wu et al. , 2018 ) as a binary classification task . For both datasets , the node features are extracted based on DeepChem ( Ramsundar et al. , 2019 ) and RDKit ( Landrum ) , which includes atom type , degree , and many other chemical properties . Since GAT was partially motivated to work on unseen data , we consider two learning settings : transductive learning and inductive learning . In the transductive learning setting , the model can access the features of all nodes in the graph . However , only a fraction of the nodes are labeled in the training phase and the model is asked to predict the missing labels . In the inductive learning setting , we have two mutually exclusive sets of nodes separately for training and testing . The model is trained only on the features and labels of the nodes in the training set and is asked to predict the labels of the nodes in the testing set . A summary of the tasks and learning settings can be found in Table 1 . We leave more detailed information like dataset statistics , training/testing split and features in Appendix A . 4 METHODOLOGY . The introduction of multi-head attention into multi-layer GNNs poses many interesting questions , we investigate five in this paper . Q1 : In the GAT model , all nodes have different attention distributions on their incoming edges . How should we characterize the overall statistics of these learned attention distributions ? Q2 : How do attention distributions differ across different heads and layers ? Q3 : How does the choice of dataset , attention variant , and learning setting affect the learned attention ? Q4 : Is the statistics of the learned attention related to the intrinsic properties of the graph ? Q5 : How to transfer attention for further usage ? To answer Q1 , we propose multiple metrics for characterizing attention distributions . For Q2 , we examine the metrics at different layers and compare the change of them over layers . To answer Q3 , we run experiments to see how varying the dataset , attention variant and the learning setting impacts the learned attention . Previous works ( Kipf & Welling , 2017 ; Hamilton et al. , 2017 ; Veličković et al. , 2018 ) only perform transductive learning on the citation networks and inductive learning on PPI . To fill in the gap , we perform transductive learning on PPI and inductive learning on the citation networks ( see data processing strategy in Appendix B.1 ) . We show that learning tasks are largely irrelevant . To answer Q4 , we propose a new task called Meta Graph Classification which asks the model to distinguish the type of the graphs by the characteristics of the attention distributions . For Q5 , we transfer attention for graph sparsification and examine whether we can preserve enough task-related information with a significant number of edges removed .

Summary: This paper presents an empirical study of the attention mechanism in the graph attention networks (GAT).  The study reveals that the attention patterns largely depend on the dataset, on some datasets they are sharp, but on others the attention patterns are almost uniform and not so different from the uniform aggregation weights in GNNs that does not have attention.  The authors further tried to utilize these findings and attempted to do attention-based graph sparsification, and showed that they can get a similar level of performance with only a fraction of the edges in the original graph if they do the sparsification based on the attention weights.