Subgraph Attention for Node Classification and Hierarchical Graph Pooling

1 INTRODUCTION . Graphs are the most suitable way to represent different types of relational data such as social networks , protein interactions and molecular structures . Typically , A graph is represented by G = ( V , E ) , where V is the set of nodes and E is the set of edges . Further , each node vi ∈ V is also associated with an attribute ( or feature ) vector xi ∈ RD . Recent advent of deep representation learning has heavily influenced the field of graphs . Graph neural networks are developed to use the underlying graph as a computational graph and aggregate node attributes from the neighbors of a node to generate the node embeddings ( Kipf & Welling , 2017 ; Niepert et al. , 2016 ) . Different types of attribute aggregation approaches are proposed in the literature ( Hamilton et al. , 2017 ) . Attention mechanisms on graphs show promising results for both node classification ( Veličković et al. , 2018 ) and graph classification ( Lee et al. , 2019 ; 2018 ) tasks . There are different ways to compute attention mechanisms on graph . Veličković et al . ( 2018 ) compute attention between a pair of nodes in the immediate neighborhood to capture the importance of a node on the embedding of the other node by learning an attention vector . Lee et al . ( 2018 ) compute attention between a pair of nodes in the neighborhood to guide the direction of a random walk in the graph for graph classification . Lee et al . ( 2019 ) propose self attention pooling of the nodes which is then used to capture the importance of the node to generate the label of the entire graph . Most of the attention mechanisms developed in graph literature use attention to derive the importance of a node or a pair of node for different tasks . But in real world situation , calculating importance up to a pair of nodes is not adequate . Often due to the presence of a substructure in the vicinity of a node v determines its role ( or label ) in the graph , or determines the label of the entire graph . But the influence of each node individually from that substructure to the node v may not be significant . In Figure 1 , each node ( indexed from a to g ) in the small synthetic graph can be considered as an agent whose attributes determine its opinion ( 1 : positive , 0 : neutral , -1 : negative ) about 4 products . Suppose the graph can be labelled +1 only if there is a subset of connected ( by edges ) agents who jointly have positive opinion about all the product . In this case , the blue shaded connected subgraph ( a , b , c ) is important to determine the label of the graph . Please note , attention over the pairs ( Veličković et al. , 2018 ) is not enough as ( a , b ) can not make the label of the graph +1 by itself . Also multiple layers of such attention may not work as the aggregated features of a node get corrupted after the feature aggregation by the first attention layer . With this motivation , we develop a novel attention mechanism in the graph which operates in the subgraph level in the vicinity of a node . We call it subgraph attention mechanism and use it for both node classification and graph classification tasks , which we define formally next . Node classification : Given a graph G = ( V , E ) with each node vi associated with an attribute vector xi ∈ RD , and a subset of nodes Vs ⊆ V with each node vi ∈ Vs labelled with yi ∈ Ln ( set of discrete labels for the nodes of the graph ) , the task is to predict the label of a node vj ∈ Vu = V \ Vs using the structure and the node attributes of the entire graph and the node labels from Vs . Essentially , this leads to learning a function fn : V 7→ Ln for the given graph G. Graph Classification : Given a set of M graphs G = { G1 , G2 , · · · , GM } , and a subset of graphs Gs ⊆ G with each graph Gi ∈ Gs are labelled with Yi ∈ Lg ( the subscript g stands for ‘ graphs ’ ) , the task is to predict the label of a graph Gj ∈ Gu = G \ Gs using the structure of the graphs and the node attributes , and the graph labels from Gs . Again , this leads to learning a function fg : G 7→ Lg . Here , Lg is the set of discrete labels for the graphs . Contributions : In this paper , we propose a novel attention mechanism ( called subgraph attention ) for graph neural networks , which is based on the importance of a subgraph of dynamic size to determine the role of a node in the graph . To validate the efficiency of the mechanism , we propose a node classification algorithm ( referred as SubGatt ) . Further , we propose a graph classification algorithm ( referred as SubGattPool ) using subgraph attention layer . SubGattPool employs a mixture of hierarchical and global pooling strategies on graph . Thorough experimentation on real world graphs shows the merit of the proposed algorithms over the state-of-the-art . 2 RELATED WORK . A survey on network representation learning and graph neural networks can be found in Wu et al . ( 2019 ) . Here we briefly discuss some more prominent approaches for node classification and graph classification . Kipf & Welling ( 2017 ) propose a version of graph convolution network ( GCN ) which learns a weighted mean of neighbor node attributes to find the embedding of a node by minimizing the cross entropy loss for node classification . Different extensions of GCN are available in literature for inductive learning ( Hamilton et al. , 2017 ) and link prediction ( Zhang & Chen , 2018 ) . Veličković et al . ( 2018 ) propose GAT which uses attention mechanism to learn the importance of a node to determine the label of another node in the neighborhood of it in the graph convolution framework . Recently , a GCN based unsupervised approach ( DGI ) is proposed ( Veličković et al. , 2019 ) by maximizing mutual information between patch representations and corresponding high-level summaries of a graph . On the other hand , graph classification is a classical problem in machine learning . Graph kernel ( Vishwanathan et al. , 2010 ) based approaches remain to be the state-of-the-art for graph classification for long time . Yanardag & Vishwanathan ( 2015 ) propose Deep Graph Kernels which learns hidden representations of sub-structures used in graph kernels . But most of the existing graph kernels use hand-crafted features . Graph2vec ( Narayanan et al. , 2017 ) is another graph embedding technique which maximizes the likelihood of the set of subgraphs which belong to a graph . Recently , different types of graph neural networks ( GNN ) are proposed for graph classification . To go from node embeddings to a single representation for the whole graph , simple aggregation technique such as taking the average of node embeddings in the final layer of a GCN ( Duvenaud et al. , 2015 ) and more advanced deep learning architectures that operate over the sets ( Gilmer et al. , 2017 ; Zhang et al. , 2018 ) have been used . Attention based graph classification technique GAM ( Lee et al. , 2018 ) is proposed , which processes only a portion of the graph by adaptively selecting a sequence of informative nodes . DIFFPOOL ( Ying et al. , 2018 ) is a recently proposed hierarchical GNN which uses a GCN based pooling to create a set of hierarchical graphs in each level . Lee et al . ( 2019 ) propose a self attention based pooling strategy which determines the importance of a node to find the label of the graph . Inspired by CapsNet ( Sabour et al. , 2017 ) , Capsule GNN ( CapsGNN ) is proposed ( Xinyi & Chen , 2019 ) . Different extensions of GNNs , such as Ego-CNN ( Tzeng & Wu , 2019 ) and ChebyGIN ( Knyazev et al. , 2019 ) are proposed for graph classification . A theoretical framework to analyze the representational power of GNNs is developed by Xu et al . ( 2019 ) , and a neural architecture GIN is proposed . As discussed in Section 1 , attention mechanisms used in the existing GNN literature only consider self-attention or the pair-wise attention between two nodes . Recently , attention over an area for a grid structure data ( for e.g. , images ) is proposed by Li et al . ( 2019 ) and show promising results for tasks like image captioning and machine translation . With this motivation , and to fill the research gap , we propose subgraph attention which generalizes attention to a subgraph level and show its efficiency for node and graph classification in this paper . 3 SUBGRAPH ATTENTION MECHANISM . In this section , we describe the building blocks for subgraph attention layer for any arbitrary graph . A Sub-Graph attention network ( referred as SubGatt ) can be built by stacking multiple layers of subgraph attention . The input to the model is an attributed graph G = ( V , E ) , where V = { v1 , v2 , · · · , vN } is the set of N nodes and xi ∈ RD is the attribute vector of the node vi ∈ V . The output of the model is a set of node features ( or embeddings ) hi ∈ RK , ∀i ∈ [ N ] ( K is potentially different from D ) . We use [ N ] to denote the set { 1 , 2 , · · · , N } for any positive integer N . We define the immediate ( or first order ) neighborhood of a node vi as Ni = { vj | ( vi , vj ) ∈ E } . For the simplicity of notations , we assume an input graph G to be undirected for the rest of the paper , but extending it for directed graph is straightforward . 3.1 SUBGRAPH SELECTION AND SAMPLING . For each node in the graph , we aim to find the importance of the nearby subgraphs to that node . In general , subgraphs can be of any shape or size . Motivated by the prior works on graph kernels Shervashidze et al . ( 2011 ) , we choose to consider only a set of rooted subtrees as the set of candidate subgraphs . So for a node vi , any tree of the form ( vi ) , or ( vi , vj ) where ( vi , vj ) ∈ E , or ( vi , vj , vk ) where ( vi , vj ) ∈ E and ( vj , vk ) ∈ E , and so on will form the set of candidate subgraphs of vi . We restrict that maximum size ( i.e. , number of nodes ) of a subtree is T . Also note that , the node vi is always a part of any candidate subgraph for the node vi according to our design . For example , all possible subgraphs of maximum size 3 for the node a in Figure 1 are : ( a ) , ( a , b ) , ( a , d ) , ( a , f ) , ( a , b , c ) , ( a , b , f ) , ( a , b , g ) , ( a , d , e ) , ( a , f , e ) and ( a , f , b ) . Depending on the maximum size ( T ) of a rooted subtree , the number of candidate subgraphs for a node can be very large . For example , the number of rooted subgraphs for the node vi is dvi ×∑ vj∈N ( vi ) ( dvj −1 ) × ∑ vk∈N ( vj ) \ { vi } |N ( vk ) \ { vi , vj } | , where dv is the degree of a node v and T = 4 . Clearly , computing attention over these many subgraphs for each node is computationally difficult . So we employ a subgraph subsampling technique , inspired by the node subsampling techniques for network embedding ( Hamilton et al. , 2017 ) . First , we fix the number of subgraphs to sample for each node . Let the number beL . For each node in the input graph , if the total number of rooted subtrees of size T is more than ( or equal to ) L , we randomly sample L number of subtrees without replacement . If the total number of rooted subtrees of size T is less than L , we use round robin sampling ( i.e. , permute all the subtrees , picking up samples from the beginning of the list ; after consuming all the trees , again start from the beginning till we complete picking L subtrees ) . For each node , sample of subtrees remains same for one epoch of the algorithm ( explained in Section 3.2 ) and new samples are taken in each epoch . In any epoch , let us use the notation Si = { Si1 , · · · , SiL } to denote the set ( more precisely it is a multiset as subgraphs can repeat ) of sampled subgraph for the node vi .

Summary: The paper proposes a novel attention approach to graph neural networks which is applicable to both of node and graph classification. The proposed method gives an attention to a subgraph instead of a node by which importance can be controlled as a set of nodes. Further, the authors also introduce two types of attention in the hierarchical structure of the network called intra- and inter- level attention.