Learning Conjoint Attentions for Graph Neural Nets

1 Introduction . Graph neural networks ( GNNs ) have shown much success in the learning of graph structured data . Amongst these noteworthy GNNs , attention-based GNNs [ 33 ] have drawn increasing interest lately , and have been applied to solve a plethora of real-world problems competently , including node classification [ 17 , 33 ] , image segmentation [ 35 ] , and social recommendations [ 30 ] . Empirical attention mechanisms adopted by GNNs aim to leverage the node features ( node embeddings ) to compute the normalized correlations between pairs of nodes that are observed to connect . Treating normalized correlations ( attention scores/coefficients ) as the relative weights between node pairs , attention-based GNN typically performs a weighted sum of node features which are subsequently propagated to higher layers . Compared with other GNNs , especially those that aggregate node features with predefined strategies [ 1 , 17 , 19 ] , attention-based GNNs provide a dynamical way for feature aggregation , which enables highly correlated features from neighboring nodes to be propagated in the multi-layer neural architecture . Representations that embed with multi-layer correlated features are consequently learned by attention-based GNNs , and can be used for various downstream tasks . Though effective , present empirical graph attention has several shortcomings when aggregating node features . First , the computation of attention coefficients is limited solely to the correlations of internal factors , i.e. , layer-wise node features within the neural nets . External factors such as cluster structure and higher-order structural similarities , which comprise heterogeneous node-node relevance have remained underexplored to be positively incorporated into the computation of more purposeful attention scores . Second , the empirical attention heavily leaning on the node features may cause 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . over-fitting in the training stage of neural nets [ 34 ] . The predictive power of attention-based GNNs is consequently limited . To overcome the mentioned challenges , in this paper , we propose a class of generic graph attention mechanisms , dubbed here as Conjoint Attentions ( CAs ) . Given CAs , we construct Graph conjoint attention networks ( CATs ) for different downstream analytical tasks . Different from previous graph attentions , CAs are able to flexibly compute the attention coefficients by not solely relying on layerwise node embeddings , but also allowing the incorporation of purposeful interventions brought by factors external to the neural net , e.g. , node cluster embeddings . With this , CATs are able to learn representations from features that are found as significant by diverse criteria , thus increasing the corresponding predictive power . The main contributions of the paper are summarized as follows . • We propose Conjoint Attentions ( CAs ) for GNNs . Different from popular graph attentions that rely solely on node features , CAs are able to incorporate heterogeneous learnable factors that can be internal and/or external to the neural net to compute purposeful and more appropriate attention coefficients . The learning capability and hence performance of CA-based GNNs is thereby enhanced with the proposed novel attention mechanisms . • For the first time , we theoretically analyze the expressive power of graph attention layers considering heterogeneous factors for node feature aggregation , and the discriminant capacity of such attention layers , i.e. , CA layers is validated . • Given CA layers , we build and demonstrate the potential of Graph conjoint attention networks ( CATs ) for various learning tasks . The proposed CATs are comprehensively investigated on established and extensive benchmarking datasets with comparison studies to a number of state-of-the-art baselines . The notable results obtained are presented to verify and validate the effectiveness of the newly proposed attention mechanisms . 2 Related works . To effectively learn low-dimensional representations in graph structured data , many GNNs have been proposed to date . According to the ways through which GNNs define the layer-wise operators for feature aggregation , GNNs can generally be categorized as spectral or spatial [ 38 ] . Spectral GNNs-The layer-wise function for feature aggregation in spectral GNNs is defined according to the spectral representation of the graph . For example , Spectral CNN [ 3 ] constructs the convolution layer based on the eigen-decomposition of graph Laplacian in the Fourier domain . However , such layer is computationally demanding . To reduce such computational burden , several approaches adopting the convolution operators which are based on simplified or approximate spectral graph theory have been proposed . First , parameterized filters with smooth coefficients are introduced for Spectral CNN to incorporate spatially localized nodes in the graph [ 14 ] . Chebyshev expansion [ 7 ] is then introduced to approximate graph Laplacian rather than directly performing eigen-decomposition . Finally , the graph convolution filter is further simplified by only considering first or higher order of connected neighbors [ 17 , 37 ] , so as to make the convolution layer more computationally efficient . Spatial GNNs-In contrast , spatial GNNs define the convolution operators for feature aggregation by directly making use of local structural properties of the central node . The essence of spatial GNNs consequently lies in designing an appropriate function for aggregating the effect brought by the features of candidate neighbors selected based on appropriate sampling strategy . To achieve this , it sometimes requires to learn a weight matrix that accords to the node degree [ 8 ] , utilize the power of transition matrix to preserve neighbor importance [ 1 , 4 , 18 , 19 , 42 ] , extract the normalized neighbors [ 24 ] , or sample a fixed number of neighbors [ 13 , 44 ] . As representative spatial GNNs , attention-based GNNs ( GATs ) [ 12 , 33 ] have shown promising performances on various learning tasks . What makes them effective in graph learning is a result of adopting the attention mechanism , which has been successfully used in machine reading and translation [ 5 , 22 ] , and video processing [ 40 ] , to compute the node-feature-based attention scores between a central node and its one-hop neighbors ( including the central node itself ) . Then , attentionbased GNNs use the attention scores to obtain a weighted aggregation of node features which are subsequently propagated to the next layer . As a result , those neighbors possessing similar features may then induce greater impact on the center node , and meaningful representations can be inferred by GATs . Having investigated previous efforts to graph neural networks , we observe that the computation of empirical graph attentions heavily relies on layer-wise node features , while other factors , e.g. , structural properties that can be learned outside of neural net , have otherwise been overlooked . This motivates us in proposing novel attention mechanisms in this paper to alleviate the shortcomings of existing attention-based graph neural networks . 3 Graph conjoint attention networks . In this section , we elaborate the proposed Conjoint Attention mechanisms , which are the cornerstones for building layers of novel attention-based graph neural networks . Mathematical preliminaries and notations used in the paper are firstly illustrated . Then , how to construct neural layers utilizing various Conjoint Attention mechanisms are introduced . Given the formulated Conjoint Attention layers , we finally construct the Graph conjoint attention networks ( CATs ) . 3.1 Notations and preliminaries . Throughout this paper , we assume a graphG = { V , E } containingN nodes , |E| edges , and C classes ( C N ) to which the nodes belong , where V and E respectively represent the node and edge set . We use A ∈ { 0 , 1 } N×N and X ∈ RN×D to represent graph adjacency matrix and input node feature matrix , respectively . Ni denotes the union of node i and its one-hop neighbors . Wl and { hli } i=1 , ... N denote the weight matrix and features ( embeddings ) of node i at lth layer of CATs , respectively , and h0 is set to be the input feature , i.e. , X . For the nodes in Ni , their possible feature vectors form a multiset Mi = ( Si , µi ) , where Si = { s1 , ... sn } is the ground set of Mi which contains the distinct elements existing in Mi , and µi : Si → N ? is the multiplicity function indicating the frequency of occurrence of each distinct s in Mi . 3.2 Structural interventions for Conjoint Attentions . As aforementioned , the proposed Conjoint Attentions are able to make use of factors that are either internal or external to the neural net to compute new attention coefficients . Internal factors refer the layer-wise node embeddings in the GNN . While , the external factors include various parameters that can be learned outside of the graph neural net and can potentially be used to compute the attention scores . Taking the cue from cognitive science , where contextual interventions have been identified as effective external factors that may improve the attention and cognitive abilities [ 16 ] , here we refer these external structural properties as structural interventions henceforth for the computing of attention coefficients . Next , we propose a simple but effective way for CAs to capture diverse structural interventions external to the GNNs . Let Cij be some structural intervention between ith and jth node in the graph . It can be obtained with the following generic generating function : Cij = arg min φ ( C ) ij Ψ ( φ ( C ) ij , Yij ) , ( 1 ) where Ψ ( · ) represents a distance function and φ ( · ) stands for an operator transforming C to the same dimensionality of Y . Given the generic generating function in Eq . ( 1 ) , it is known that many effective paradigms for learning latent features can be used for the subsequent computation of Conjoint Attentions , if the prior feature matrix Y is appropriately provided . Taking A as the prior feature matrix , in this paper , we consider two generation processes that can capture two unique forms of structural interventions . Let Ψ ( · ) be the euclidean distance , when φ ( C ) ij . = VVTij , we have : Cij = arg min VVTij ( Aij −VVTij ) 2 , ( 2 ) where we use an N -by-C matrix V to approximate C to reduce the computational burden . As it shows in Eq . ( 2 ) , Cij attempts to acquire the structural correlation pertaining to node cluster embeddings , based on matrix factorization ( MF ) . A higher Cij learned by Eq . ( 2 ) means a pair of nodes are very likely to belong to the same cluster . If φ ( C ) ij . = ∑ jVV T ijAij , we have : Cij = arg min VVTij ( Aij − ∑ j VVTijAij ) 2 . ( 3 ) As shown in Eq . ( 3 ) , Cij is the coefficient of self-expressiveness [ 10 ] ( SC ) which may describe the global relation between node i and j . A higher Cij inferred by Eq . ( 3 ) means the global structure of node i can be better represented by that of node j , and consequently this pair of nodes are more structurally correlated . It is known that both aforementioned properties have not been considered previously by empirical graph attention mechanisms . We believe considering either of them as structural interventions for the Conjoint Attentions could lead to better attention scores for feature aggregation . Note that other types of C may also be feasible for the proposed attention mechanisms , as long as they are able to capture meaningful property which is not already possessed within the node embeddings of the GNN .

Summary: The authors propose Graph ConJoint Attention Networks (CATs) as a mechanism for explicitly incorporating structural information in the GATs attention mechanism.  CATs separately learn an "enlightenment" attention matrix which exploits structural correlations between the vertices (based on the adjacency matrix).  Such a matrix is then recombined with the attention matrix computed by vanilla GATs, yielding updated attentional coefficients which empirically improve on GATs on several benchmark datasets, and can be proven (with small modifications) to satisfy the equivalence with the 1-WL test, which also answers a somewhat-open question on GATs' expressive power.