Graph Joint Attention Networks

1 INTRODUCTION . Many real-world data can be modeled as a graph , where a set of nodes ( vertices ) , edges , and bag-ofwords features respectively represent data instances , instance-instance interrelationships , and contents characterizing the nodes . For example , scientific articles in a research domain can be modeled as a graph , where nodes , edges , and node features respectively represent published articles , citations , and index information of the articles . Besides , social network users and interacted biological units can also be similarly represented as graphs possessing different structural and descriptive information . As graph data are widely available and they are related to various analytical tasks , learning in graphs has been a hot-spot in machine learning community . There have been a number of approaches proposed to effectively learn in graph structured data . Amongst them , graph convolutional networks ( GCNs ) have shown to be powerful in learning lowdimensional representations for various subsequent analytical tasks . Different from those empirical convolutional neural networks ( CNNs ) which have achieved a great success in learning in image , vision , and natural language data ( Krizhevsky et al. , 2012 ; Xu et al. , 2014 ) , and whose convolution operators are always defined to process a grid-like data structure , GCNs attempt to formulate convolution operators aggregating the node features according to the observed graph structure , and learn the information propagation through different neural architectures . Meaningful representations which capture discriminative node features as well as intricate graph structure can thereby be learned by GCNs . There have been several sophisticated GCNs proposed in the recent . According to the ways through which GCNs make use of graph topology to define convolution operators for feature aggregation , GCNs can generally be categorized as spectral , and spatial ones ( Wu et al. , 2020 ) . Spectral GCNs define the convolutional layer for aggregating neighbor features based on the spectral representation of the graph . For example , Spectral CNN ( Bruna et al. , 2013 ) constructs the convolution layer based on the eigen-decomposition of graph Laplacian in the Fourier domain . However , such layer is computationally demanding . Aiming to reduce such computational burden , several approaches adopting the convolution operators which are based on simplified/approximate spectral graph theory are proposed . First , parameterized filters with smooth coefficients are introduced for Spectral CNN to allow it to consider spatially localized nodes in the graph ( Henaff et al. , 2015 ) . Chebyshev expansion is then introduced in ( Defferrard et al. , 2016 ) to approximate graph Laplacian rather than directly perform eigen-decomposition of it . Finally , the graph convolution filter is further simplified by only considering connected neighbors of each node ( Kipf & Welling , 2017 ) so as to further make spectral GCNs computationally efficient . In contrast , spatial GCNs define the convolution operators for feature aggregation directly making use of local structural properties of the central node . The key of spatial GCNs is consequently how to design an appropriate function for aggregating the effect brought by the features of candidate neighbors selected according to a proper sampling strategy . To achieve this , it sometimes requires to learn a weight matrix according to node degree ( Duvenaud et al. , 2015 ) , utilize the power of transition matrix to preserve the neighbor importance ( Atwood & Towsley , 2016 ; Busch et al. , 2020 ; Klicpera et al. , 2019 ) , extract the normalized neighbors ( Niepert et al. , 2016 ) , or sample a fixed number of neighbors ( Hamilton et al. , 2017 ; Zhang et al. , 2020 ) . As a representative spatial GCN , Graph attention network ( GAT ) ( Veličković et al. , 2018 ; Zhang et al. , 2020 ) has shown a promising performance in various graph learning tasks . What makes GATs effective in learning graph representations is they adopt the attention mechanism , which has been successfully used in machine reading and translation ( Luong et al. , 2015 ; Cheng et al. , 2016 ) , and video processing ( Xu et al. , 2015 ) , to compute the node-feature-based attention weights ( attention scores ) between a central node and its one-hop neighbors ( including the central node itself ) . Then , GATs utilize the attention scores to obtain a weighted aggregation of node features which are propagated to the next layer . As a result , those neighbors possessing similar features may impact more on the center node , and meaningful representations can be inferred by GATs . Although GATs have been experimentally verified as powerful tools for various graph learning tasks , they still confront several challenges . First , for attention-based GNNs , appropriate attention mechanisms which can automatically identify the relative significance between the graph structure and node features are not many . As a result , most current attention mechanisms for GATs can not effectively capture the joint effect brought by the underlying graph structure and node features for seamlessly impacting the message-passing in the neural architecture . Second , whether the expressive power of GNNs adopting the attention mechanisms which can effectively acquire the aforementioned joint effect may reach the upper bound of message-passing GNNs has not been theoretically investigated . To address the mentioned challenges , in this paper , we propose novel attention-based GNNs , dubbed Graph Joint Attention Networks ( JATs ) . Different from previous works , the attention mechanisms adopted by JATs are able to automatically capture the relative significance between structural coefficient learned from graph topology , and node features , so that higher attention scores may be learned by those neighbors which are topologically and contextually correlated . JATs are consequently able to smoothly adjust attention scores according to the contemporary structure and node features , and truly capture the joint attention on structural and contextual information propagated in the neural network . Besides , we theoretically analyze the expressive power of JATs and further propose an improved strategy which enables JATs to distinguish all distinct graph structures as 1-dimensional Weisfeiler-Lehman test ( 1-WL test ) does . This means JATs can reach the upper bound w.r.t . expressive power which all message-passing GNNs can ultimately achieve . JATs have been extensively tested on four widely used datasets , i.e. , Cora , Citeseer , Pubmed , and OGBNArxiv , and have been compared with a number of strong baselines . The experimental results show that JATs achieve the state-of-the-art performance . The rest of the paper is organized as follows . In Section 2 , we elaborate the proposed JATs , and compare JATs with other GNNs . In Section 3 , we prove the limitation w.r.t . expressive power of the joint attention mechanisms presented in Section 2 . A strategy is then proposed to improve JATs to reach the upper bound of expressive power which all message-passing GNNs can at most achieve . The comprehensive experiments which are used to validate the effectiveness of JATs are presented in Section 4 . Finally , we summarize the contributions of the paper and propose future works potentially improving JATs . 2 JOINT ATTENTION-BASED GRAPH NEURAL NETWORKS . In this section , we elaborate the proposed JATs . Mathematical preliminaries and notations used in the paper are firstly illustrated . How JATs learn the structural coefficients which are used in the joint attention mechanisms is then introduced . Following that , the joint attention layer , which is the cornerstone of JATs is elaborated . At last , we compare the proposed JATs with their counterparts . 2.1 NOTATIONS AND PRELIMINARIES . Throughout this paper , we assume a graph G = { V , E } containing N nodes , |E| edges , where V and E respectively represent the node and edge set . We use A ∈ { 0 , 1 } N×N and X ∈ RN×D to represent graph adjacency matrix and node feature matrix , respectively . Ni denotes the union of node i and its one-hop neighbors . Wl and { hli } i=1 , ... N denote the weight matrix and features of node i at lth layer of JATs , respectively , and h0 is set to be the input feature , i.e. , X . For the nodes in Ni , their possible feature vectors form a multiset Mi = ( Si , µi ) , where Si = { s1 , ... sn } is the ground set of Mi which contains the distinct elements existing in Mi , and µi : Si → N ? is the multiplicity function indicating the frequency of occurrence of each distinct s in Mi . 2.2 LEARNING STRUCTURAL COEFFICIENTS FROM GRAPH SUBSPACE . It is well known that topology is the corner stone of the graph . How to utilize such structural information to compute the attention scores naturally has a profound impact on the performance of attention-based GNNs . Empirical attention based GNNs , e.g. , GAT , compute the attention scores between connected neighbors only using node features , but overlook the structural correlation between pairwise nodes . To allow attention-based GNNs to capture the higher-order structures in the graph , we propose JATs to learn the topological coefficients from the graph subspace . Inspired by subspace clustering ( Elhamifar & Vidal , 2013 ) , we may formulate the learning of structural coefficients as follows . Given N nodes in the graph drawn from multiple linear subspaces { Si } i=1 , ... K , one can represent a node in a subpace as a linear combination of other nodes . If each row in A is treated as the structural information of each node , one can simply represent it using other nodes ( other rows in A ) as one equation , i.e. , Ai , : = Ci , :A , where C denotes the structural coefficient matrix as A is used . It has been shown in previous works ( Ji et al. , 2014 ) that under the assumption the subsapces are independent , by minimizing certain norm of C , C may have a block diagonal structure ( after finite permutations ) . In other words , Cij 6= 0 if and only if two nodes , vi and vj are in the same subspace . So , we can utilize C to learn the structural correlations between neighbors in the graph . And the above learning task can be formulated as the following optimization problem : minimize ‖C‖p , subject to A = CA , Cii = 0 , ( 1 ) where ‖·‖p stands for a certain matrix norm , and the zero constraint on the diagonal of C may prevent trivial solutions when ‖·‖p is the norm considering sparsity . To make the data corruption explainable , the equality constraint in Eq . ( 1 ) is often relaxed as a regularization term and the learning of structural coefficients can be reformulated as follows : minimize ‖C‖p + β‖A−CA‖2F , subject to Cii = 0 . ( 2 ) By rewriting Eq . ( 2 ) , we may reveal why subspace learning is effective in capturing the structural correlations between pairwise nodes . Mathematically , Eq . ( 2 ) is equivalent to the following nodewise minimization problem : minimize ∑ i ‖Ci , :‖p + β‖Ai , : −CAi , :‖2F , subject to Cii = 0 . ( 3 ) As CAi , : is equal to ∑ j CijAj , : and Cii = 0 , one may easily find that minimizing Eq . ( 3 ) is equivalent to search for the optimal linear combination of other nodes that can be used to reconstruct the ith node in the graph . As a result , Cij is high when Aj , : , i.e. , the global structure of jth node ( rather than the local bias ( Gong et al. , 2018 ) ) , is similar to Ai , : . By minimizing Eq . ( 2 ) , one may identify those nodes which are in the same graph subspace , and the structural correlations between neighboring nodes can therefore be inferred directly . As the above learning problem can be effectively solved via gradient descent , JATs can optimize Eq . ( 2 ) together with the training of the neural architecture . Also , we use l1 norm for C to force JATs to learn sparse structural coefficients .

Summary: Creating aggregation weight over neighbor nodes lies at the key part of graph neural networks. Generally the weights can be generated by the node structure or feature similarity. The node structure similarity provides a way to measure the correlation of a pair of nodes with a complete graph. While attention weight usually focuses on the local neighborhood, which can be easily biased by the node popularity [1, 2]. If a neighbor node with large degree, it potentially tends to have a large embedding norm. However, from complete view of graph structure, this neighbor might be a noisy node. With a given complete graph structure, we can downgrade the influence of structure bias. From this point of view, it's interesting to study whether the attention and structure weights can be complementary to each other. The key idea of this work is very easy to follow. The proposed strategy is to unify the weights generated by graph structure and node feature. Theoretical analysis shows the expressive power of  the proposed strategy. The experimental results for node classification also demonstrates its superiority comparing with state-of-the-art baselines.