Diagonal Graph Convolutional Networks with Adaptive Neighborhood Aggregation

1 INTRODUCTION . The past decade has witnessed the remarkable success of deep neural networks across different domains , ranging from computer vision ( Krizhevsky et al. , 2012 ; He et al. , 2016 ) , to speech recognition ( Abdel-Hamid et al. , 2014 ) , to natural language processing ( Gehring et al. , 2017 ) . These breakthroughs have provoked the interest to extend neural operations from linear or spatial domains to arbitrary graph structures , e.g. , social networks , gene-protein networks , and knowledge graphs . One challenge in learning with graph-structured data is how to represent each node or subgraph in neural networks . This is challenging , as graph data lies in the non-Euclidean space with complex structures and random size , making it infeasible to directly apply traditional neural network algorithms ( Hamilton et al. , 2017 ; Kipf & Welling , 2016 ) . Recently , there have been attempts to generalize neural networks for handling graph-structured data . For example , graph neural networks ( GNNs ) are introduced as recursive neural networks in which node features are propagated iteratively until equilibrium ( Gori et al. , 2005 ; Scarselli et al. , 2009 ) . More recently , several studies show that it is natural for convolutional neural networks ( CNNs ) to model graph data structure by defining spatial localized convolutional filters over graph Laplacian spectral space ( Bruna et al. , 2013 ; Henaff et al. , 2015 ; Defferrard et al. , 2016 ) or more directly , over graph neighborhoods ( Duvenaud et al. , 2015 ; Niepert et al. , 2016 ; Monti et al. , 2017 ) . These techniques , named as graph convolutional networks ( GCNs ) ( Kipf & Welling , 2016 ) , enable deep neural networks to effectively capture graph topology and properties . Over the past three years , various GCN based models , such as GraphSAGE ( Hamilton et al. , 2017 ) , FastGCN ( Chen et al. , 2018 ) , and GAT ( Velickovic et al. , 2018 ) , have been rapidly developed , offering promising results for important graph mining tasks , such as , node classification and link prediction . In theory , GCNs can be considered as a simplification of the traditional graph spectral methods ( Kipf & Welling , 2016 ) . The common strategy is to model a node ’ s neighborhood as the receptive field and then define the “ graph convolution layer ” as a mean pooling aggregation followed by a nonlinearity encoder . Existing GCNs stack multiple layers and learn node representations by recursively aggregating information from neighbors , and the analysis of their representational capacity can be found in a recent study ( Xu et al. , 2019 ) . Essentially , the recursive convolutional aggregation process in GCNs , or the Neural Message Passing framework ( Gilmer et al. , 2017 ) , is also closely related to the Weisfeiler-Lehman ( WL ) graph isomorphism test ( Weisfeiler & Lehman , 1968 ) . Though GCNs have been extensively explored , how to better capture the graph structures through convolutions is still largely an open question . Particularly , the neighborhood aggregation step in GCNs can be seen as the diffusion process in real-world networks ( Gilmer et al. , 2017 ) , in which different nodes play different roles in the propagation . For example , for information diffusion in social networks , opinion leaders ( Granovetter , 1973 ) are more likely to expose information to associated communities and structural holes ( Burt , 2009 ) can spread information across different groups . However , the neighborhood aggregation step in most existing GCNs considers neither the different structural importance of nodes within each step nor the dynamics of node importance across different propagation steps ( Kipf & Welling , 2016 ; Hamilton et al. , 2017 ; Chen et al. , 2018 ) . In light of this limitation , we propose an adaptive neighborhood aggregation strategy to model and aggregate the dynamics of node importance . The adaptive neighborhood aggregation step consists of two coupled operations : node adaptive rescaling and node adaptive encoder . The idea of node adaptive rescaling is to adaptively learn each node ’ s importance and use it to automatically rescale the output signal of the original graph convolutional operator ; In addition to this customized rescaling , each node can also adaptively choose its encoder function according to its importance , forming the concept of node adaptive encoder . With these two techniques in neighborhood aggregation , we present the diagonal graph convolutional networks ( DiagGCN ) , which is able to dynamically depict and model nodes ’ importance in GCNs . Importantly , there are several advantages of the proposed DiagGCN model . First , efficiency wise , the adaptive rescaling is achieved by a simple diagonal matrix operation , making DiagGCN ’ s complexity the same as the sparse version of GCNs , i.e. , linear with the network volume . In addition , the rescaling step is also parallelizable across all nodes in the graph . Second , it naturally supports both transductive and inductive learning , meaning that the model can generalize inference to unseen nodes . Third , DiagGCN can be connected with graph attention with different mechanisms , including node attention , ( multi-hop ) edge attention , and path attention , providing a new perspective to understand graph attention . This flexible framework potentially offers a principled way to automatically choose good attention scheme , or reasonable permutation on the adjacency matrix , rescaling diagonal matrices , and adaptive encoders ( i.e. , AutoML for graph convolution/attention ) . Specifically , we show the GAT model ( Velickovic et al. , 2018 ) can be reformulated as a special case of the proposed DiagGCN model with edge attention . Finally , our extensive experiments on several graph benchmark datasets of different types , including Cora , Citeseer , Pubmed citation networks , and the protein-protein interaction network ( PPI ) show that the proposed framework consistently achieves performance superiority over existing state-of-art GCN based baselines . 2 DEFINITIONS AND PRELIMINARIES . Let G = ( V , E , X ) denote a graph , where V is a set of |V | = n nodes , E is a set of |E| = m edges between nodes , and X ∈ Rn×d is a feature matrix recording d features associated with each node . Further let A ∈ { 0 , 1 } n×n denote the adjacency matrix of the graph G , with an element Aij > 0 indicating node vi ∈ V has a link to vj ∈ V . Our node representation framework is built upon graph convolutional networks . Here , we refer to the definition of graph convolution in spectral methods and it can be fast approximated by the multilayer graph convolution networks ( GCNs ) ( Kipf & Welling , 2016 ) with the following layer-wise propagation rule : H ′ = σ ( ÂHW ) ( 1 ) where Â = D̃− 1 2 Ã D̃− 1 2 is the normalized adjacency matrix , Ã = A + In is the adjacency matrix with augmented self-connections , In is the identity matrix , D̃ is the diagonal degree matrix with D̃ii = ∑ j Ãij , andW is a layer-specific learnable weight matrix . Function σ ( . ) denotes a nonlinear activation function . H ∈ Rn×D is the matrix of activations , or the D-dimensional hidden node representation in the corresponding layer with H ( 0 ) = X . 3 THE DIAGONAL GRAPH CONVOLUTIONAL NETWORKS . In observation of different importance of nodes in real network propagation processes , we present a new graph convolution neural network framework ( DiagGCN ) with an adaptive neighborhood aggregation rule . The rule is coupled with the node adaptive information rescaling and encoder , where every node can process the graph information distinctly . We also show that the information processing capabilities of different nodes in different propagation steps can be formally implemented with a series of diagonal matrices . Further , sophisticated propagation models ( e.g. , the graph attention networks with node attention , edge attention , and path attention ) can also be formulated as the permutations of the adjacency matrix and diagonal matrices in the DiagGCN model . 3.1 BASIC IDEA . In general , our proposed model , DiagGCN , where Diag stands for diagonal matrix , is mainly stacked by the following building block : H ′ i = σi [ ( PÂQHW ) i ] ( 2 ) In each layer , compared with the original GCNs , we only add a diagonal matrix P and a diagonal matrix Q to the left and right sides of the normalized adjacency matrix Â , respectively . Importantly , all Pii , Qii , and the node-wise activation function σi are adaptively adjusted according to the input information on node i , and their concrete forms will be introduced in Sections 3.2 and 3.3 . Intuition . Complex propagation or diffusion processes over a network are driven by the diversity of nodes ’ status and influence ( Kempe et al. , 2003 ) . Nodes play different roles and process the input information signal in different ways . For example , the structural hole nodes may amplify the information across different groups of nodes in social contagion ( Burt , 2009 ) , and opinion leader nodes are more likely to have a broad spread of information to the public , thus may amplify most kinds of their information ( Katz , 1957 ) . However , in most existing GCN models ( Kipf & Welling , 2016 ; Hamilton et al. , 2017 ; Chen et al. , 2018 ) , the neighborhood aggregation step can be considered as a simple diffusion process over networks , by ignoring the difference between nodes . To this end , we propose to bridge the gap between GCNs ’ simple neighborhood aggregation and real-world network diffusion . The proposed DiagGCN model realizes different ways of processing information on different nodes through the node adaptive rescaling factor and encoder functions . In other words , each node has an adaptive information gate , which is also simpler than sophisticated GCN models with parameterized adjacency matrices , such as MoNet ( Monti et al. , 2017 ) and GAT ( Velickovic et al. , 2018 ) . Furthermore , the permutations of the adjacency matrix and diagonal matrices in DiagGCN offer a general and flexible framework to design powerful GCNs . Computationally , DiagGCN adds the diagonal matrices to the original GCN architecture , thus the computational complexity is also linear with the volume of the network , which is the same as the sparse version of GCNs . Meanwhile , the computation of node adaptive rescaling factors can be parallelized across all nodes . 3.2 NODE ADAPTIVE RESCALING . In Eq . 2 , we use the diagonal matrix Q to rescale the rows of the signal HW from the left . In particular , the signal ( HW ) i on node i is rescaled by the factor Qii adaptively adjusted by node i . Figure 1 illustrates the rescaling process . Basically , we use a learnable vector ~s1 to denote the current network state , and the importance of node i is the dot product similarity of its signal ( HW ) i and the network state ~s1 , that is , ri = ~s T 1 ( HW ) i ( 3 ) and the rescaling factor is the sigmoid function ( or other monotonic like softplus and exponential function ) of the node importance , i.e. , Qii = sigmoid ( ri ) ( 4 ) Dropout mechanism is also applied on Qii to randomly removing the neighbor node in the aggregation step when training . After the rescaling of Q and the propagation of Â , the signal of node i is transformed as ( ÂQHW ) i . We then use the diagonal matrix P , which is defined below , to renormalize the signals , i.e. , Pii = 1∑ j ÂijQjj ( 5 ) Note that P can also have the same rescaling form asQ . We will show later how different definitions of P can be characterized as various graph attention mechanisms .

Summary: This paper proposed  a so-called diagonal GCN method with adaptive neighborhood aggregation rules., namely, each node will be associated with an individual importance factor to re-scale the output signal  of the graph convolutional operator, and also an adaptive encoder function is adopted. This is achieved by adding a left diagonal matrix P and a right diagonal matrix Q to the standard message passing of GCN H’ = sigma(AHW), as H’(PAQHW); P and Q are parameterized based on the HW part (node features). Multihead attention and multi-hop versions are also derived. Connection of this formulation to the existing GAT method is also discussed. Empirical results on transductive node classification and inductive node classification are reported against state-of-the-arts.