Non-Local Graph Neural Networks

1 INTRODUCTION . Graph neural networks ( GNNs ) process graphs and map each node to an embedding vector ( Zhang et al. , 2018b ; Wu et al. , 2019 ) . These node embeddings can be directly used for node-level applications , such as node classification ( Kipf & Welling , 2017 ) and link prediction ( Schütt et al. , 2017 ) . In addition , they can be used to learn the graph representation vector with graph pooling ( Ying et al. , 2018 ; Zhang et al. , 2018a ; Lee et al. , 2019 ; Yuan & Ji , 2020 ) , in order to fit graph-level tasks ( Yanardag & Vishwanathan , 2015 ) . Many variants of GNNs have been proposed , such as ChebNets ( Defferrard et al. , 2016 ) , GCNs ( Kipf & Welling , 2017 ) , GraphSAGE ( Hamilton et al. , 2017 ) , GATs ( Veličković et al. , 2018 ) , LGCN ( Gao et al. , 2018 ) and GINs ( Xu et al. , 2019 ) . Their advantages have been shown on various graph datasets and tasks ( Errica et al. , 2020 ) . However , these GNNs share a multilayer local aggregation framework , which is similar to convolutional neural networks ( CNNs ) ( LeCun et al. , 1998 ) on grid-like data such as images and texts . In recent years , the importance of non-local aggregation has been demonstrated in many applications in the field of computer vision ( Wang et al. , 2018 ; 2020 ) and natural language processing ( Vaswani et al. , 2017 ) . In particular , the attention mechanism has been widely explored to achieve non-local aggregation and capture long-range dependencies from distant locations . Basically , the attention mechanism measures the similarity between every pair of locations and enables information to be communicated among distant but similar locations . In terms of graphs , non-local aggregation is also crucial for disassortative graphs , while previous studies of GNNs focus on assortative graph datasets ( Section 2.2 ) . In addition , we find that local aggregation is even harmful for some disassortative graphs ( Section 4.3 ) . The recently proposed Geom-GCN ( Pei et al. , 2020 ) explores to capture longrange dependencies in disassortative graphs . It contains an attention-like step that computes the Euclidean distance between every pair of nodes . However , this step is computationally prohibitive for large-scale graphs , as the computational complexity is quadratic in the number of nodes . In addition , Geom-GCN employs pre-trained node embeddings ( Tenenbaum et al. , 2000 ; Nickel & Kiela , 2017 ; Ribeiro et al. , 2017 ) that are not task-specific , limiting the effectiveness and flexibility . In this work , we propose a simple yet effective non-local aggregation framework for GNNs . At the heart of the framework lies an efficient attention-guided sorting , which enables non-local aggregation through classic local aggregation operators in general deep learning . The proposed framework can be flexibly used to augment common GNNs with low computational costs . Based on the framework , we build various efficient non-local GNNs . In addition , we perform detailed analysis on existing disassortative graph datasets , and apply different non-local GNNs accordingly . Experimental results show that our non-local GNNs significantly outperform previous state-of-the-art methods on node classification tasks on six benchmark datasets of disassortative graphs . 2 BACKGROUND AND RELATED WORK . 2.1 GRAPH NEURAL NETWORKS . We focus on learning the embedding vector for each node through graph neural networks ( GNNs ) . Most existing GNNs are inspired by convolutional neural networks ( CNNs ) ( LeCun et al. , 1998 ) and follow a local aggregation framework . In general , each layer of GNNs scans every node in the graph and aggregates local information from directly connected nodes , i.e. , the 1-hop neighbors . Specifically , a common layer of GNNs performs a two-step processing similar to the depthwise separable convolution ( Chollet , 2017 ) : spatial aggregation and feature transformation . The first step updates each node embedding using embedding vectors of spatially neighboring nodes . For example , GCNs ( Kipf & Welling , 2017 ) and GATs ( Veličković et al. , 2018 ) compute a weighted sum of node embeddings within the 1-hop neighborhood , where weights come from the degree of nodes and the interaction between nodes , respectively . GraphSAGE ( Hamilton et al. , 2017 ) applies the max pooling , while GINs ( Xu et al. , 2019 ) simply sums the node embeddings . The feature transformation step is similar to the 1×1 convolution , where each node embedding vector is mapped into a new feature space through a shared linear transformation ( Kipf & Welling , 2017 ; Hamilton et al. , 2017 ; Veličković et al. , 2018 ) or multilayer perceptron ( MLP ) ( Xu et al. , 2019 ) . Different from these studies , LGCN ( Gao et al. , 2018 ) explores to directly apply the regular convolution through top-k ranking . Nevertheless , each layer of these GNNs only aggregates local information within the 1-hop neighborhood . While stacking multiple layers can theoretically enable communication between nodes across the multi-hop neighborhood , the aggregation is essentially local . In addition , deep GNNs usually suffer from the over-smoothing problem ( Xu et al. , 2018 ; Li et al. , 2018 ; Chen et al. , 2020 ) . 2.2 ASSORTATIVE AND DISASSORTATIVE GRAPHS . There are many kinds of graphs in the literature , such as citation networks ( Kipf & Welling , 2017 ) , community networks ( Chen et al. , 2020 ) , co-occurrence networks ( Tang et al. , 2009 ) , and webpage linking networks ( Rozemberczki et al. , 2019 ) . We focus on graph datasets corresponding to the node classification tasks . In particular , we categorize graph datasets into assortative and disassortative ones ( Newman , 2002 ; Ribeiro et al. , 2017 ) according to the node homophily in terms of labels , i.e. , how likely nodes with the same label are near each other in the graph . Assortative graphs refer to those with a high node homophily . Common assortative graph datasets are citation networks and community networks . On the other hand , graphs in disassortative graph datasets contain more nodes that have the same label but are distant from each other . Example disassortative graph datasets are co-occurrence networks and webpage linking networks . As introduced above , most existing GNNs perform local aggregation only and achieve good performance on assortative graphs ( Kipf & Welling , 2017 ; Hamilton et al. , 2017 ; Veličković et al. , 2018 ; Gao et al. , 2018 ) . However , they may fail on disassortative graphs , where informative nodes in the same class tend to be out of the local multi-hop neighborhood and non-local aggregation is needed . Thus , in this work , we explore the non-local GNNs . 2.3 ATTENTION MECHANISM . The attention mechanism ( Vaswani et al. , 2017 ) has been widely used in GNNs ( Veličković et al. , 2018 ; Gao & Ji , 2019 ; Knyazev et al. , 2019 ) as well as other deep learning models ( Yang et al. , 2016 ; Wang et al. , 2018 ; 2020 ) . A typical attention mechanism takes three groups of vectors as inputs , namely the query vector q , key vectors ( k1 , k2 , . . . , kn ) , value vectors ( v1 , v2 , . . . , vn ) . Note that key and value vectors have a one-to-one correspondence and can be the same sometimes . The attention mechanism computes the output vector o as ai = ATTEND ( q , ki ) ∈ R , i = 1 , 2 , . . . , n ; o = ∑ i aivi , ( 1 ) where the ATTEND ( · ) function could be any function that outputs a scalar attention score ai from the interaction between q and ki , such as dot product ( Gao & Ji , 2019 ) or even a neural net- work ( Veličković et al. , 2018 ) . The definition of the three groups of input vectors depends on the models and applications . Notably , existing GNNs usually use the attention mechanism for local aggregation ( Veličković et al. , 2018 ; Gao & Ji , 2019 ) . Specifically , when aggregating information for node v , the query vector is the embedding vector of v while the key and value vectors come from node embeddings of v ’ s directly connected nodes . And the process is iterated for each v ∈ V . It is worth noting that the attention mechanism can be easily extended for non-local aggregation ( Wang et al. , 2018 ; 2020 ) , by letting the key and value vectors correspond to all the nodes in the graph when aggregating information for each node . However , it is computationally prohibitive given large-scale graphs , as iterating it for each node in a graph of n nodes requires O ( n2 ) time . In this work , we propose a novel non-local aggregation method that only requires O ( n log n ) time . 3 THE PROPOSED METHOD . 3.1 NON-LOCAL AGGREGATION WITH ATTENTION-GUIDED SORTING . We consider a graph G = ( V , E ) , where V is the set of nodes and E is the set of edges . Each edge e ∈ E connects two nodes so that E ⊆ V ×V . Each node v ∈ V has a node feature vector xv ∈ Rd . The k-hop neighborhood of v refers to the set of nodes Nk ( v ) that can reach v within k edges . For example , the set of v ’ s directly connected nodes is its 1-hop neighborhood N1 ( v ) . Our proposed non-local aggregation framework is composed of three steps , namely local embedding , attention-guided sorting , and non-local aggregation . In the following , we describe them one by one . Local Embedding : Our proposed framework is built upon a local embedding step that extracts local node embeddings from the node feature vectors . The local embedding step can be as simple as zv = MLP ( xv ) ∈ Rf , ∀v ∈ V. ( 2 ) The MLP ( · ) function is a multilayer perceptron ( MLP ) , and f is the dimension of the local node embedding zv . Note that the MLP ( · ) function is shared across all the nodes in the graph . Applying MLP only takes the node itself into consideration without aggregating information from the neighborhood . This property is very important on some disassortative graphs , as shown in Section 4.3 . On the other hand , graph neural networks ( GNNs ) can be used as the local embedding step as well , so that our proposed framework can be easily employed to augment existing GNNs . As introduced in Section 2.1 , modern GNNs perform multilayer local aggregation . Typically , for each node , one layer of a GNN aggregates information from its 1-hop neighborhood . Stacking L such local aggregation layers allows each node to access information that is L hops away . To be specific , the ` -th layer of a L-layer GNN ( ` = 1 , 2 , . . . , L ) can be described as z ( ` ) v = TRANSFORM ( ` ) ( AGGREGATE ( ` ) ( { z ( ` −1 ) u : u ∈ N1 ( v ) ∪ v } ) ) ∈ Rf , ∀v ∈ V , ( 3 ) where z ( 0 ) v = xv , and zv = z ( L ) v represents the local node embedding . The AGGREGATE ( ` ) ( · ) and TRANSFORM ( ` ) ( · ) functions represent the spatial aggregation and feature transformation step introduced in Section 2.1 , respectively . With the above framework , GNNs can capture the node feature information from nodes within a local neighborhood as well as the structural information . When either MLP or GNNs is used as the local embedding step , the local node embedding zv only contains local information of a node v. However , zv can be used to guide non-local aggregation , as distant but informative nodes are likely to have similar node features and local structures . Based on this intuition , we propose the attention-guided sorting to enable the non-local aggregation . Attention-Guided Sorting : The basic idea of the attention-guided sorting is to learn an ordering of nodes , where distant but informative nodes are put near each other . Specifically , given the local node embedding zv obtained through the local embedding step , we compute one set of attention scores by av = ATTEND ( c , zv ) ∈ R , ∀v ∈ V , ( 4 ) where c is a calibration vector that is randomly initialized and jointly learned during training ( Yang et al. , 2016 ) . In this attention operator , c serves as the query vector and zv are the key vectors . In addition , we also treat zv as the value vectors . However , unlike the attention mechanism introduced in Section 2.3 , we use the attention scores to sort the value vectors instead of computing a weighted sum to aggregating them . Note that originally there is no ordering among nodes in a graph . To be specific , as av and zv have one-to-one correspondence through Equation ( 4 ) , sorting the attention scores in non-decreasing order into ( a1 , a2 , . . . , an ) provides an ordering among nodes , where n = |V | is the number of nodes in the graph . The resulting sequence of local node embeddings can be denoted as ( z1 , z2 , . . . , zn ) . The attention process in Equation ( 4 ) can be also understood as a projection of local node embeddings onto a 1-dimensional space . The projection depends on the concrete ATTEND ( · ) function and the calibration vector c. As indicated by its name , the calibration vector c is used to calibrate the 1-dimensional space , in order to push distant but informative nodes close to each other in this space . This goal is fulfilled through the following non-local aggregation step and the training of the calibration vector c , as demonstrated below . Non-Local Aggregation : We point out that , with the attention-guided sorting , the non-local aggregation can be achieved by convolution , the most common local aggregation operator in deep learning . Specifically , given the sorted sequence of local node embeddings ( z1 , z2 , . . . , zn ) , we compute ( ẑ1 , ẑ2 , . . . , ẑn ) = CONV ( z1 , z2 , . . . , zn ) , ( 5 ) where the CONV ( · ) function represents a 1D convolution with appropriate padding . Note that the CONV ( · ) function can be replaced by a 1D convolutional neural network as long as the number of input and output vectors remains the same . To see how the CONV ( · ) function performs non-local aggregation with the attention-guided sorting , we take an example where the CONV ( · ) function is a 1D convolution of kernel size 2s + 1 . In this case , ẑi is computed from ( zi+s , . . . , zi−s ) , corresponding to the receptive field of the CONV ( · ) function . As a result , if the attention-guided sorting leads to ( zi+s , . . . , zi−s ) containing nodes that are distant but informative to zi , the output ẑi aggregates non-local information . Another view is that we can consider the attention-guided sorting as re-connects nodes in the graph , where ( zi+s , . . . , zi−s ) can be treated as the 1-hop neighborhood of zi . After the CONV ( · ) function , ẑi and zi are concatenated as the input to a classifier to predict the label of the corresponding node , where both non-local and local dependencies can be captured . In order to enable the end-to-end training of the calibration vector c , we modify Equation ( 5 ) into ( ẑ1 , ẑ2 , . . . , ẑn ) = CONV ( a1z1 , a2z2 , . . . , anzn ) , ( 6 ) where we multiply the attention score with the corresponding local node embedding . As a result , the calibration vector c receives gradients through the attention scores during training . The remaining question is how to make sure that the attention-guided sorting pushes distant but informative nodes together . The short answer is that it is not necessary to guarantee this , as the requirement of non-local aggregation depends on the concrete graphs . In fact , our proposed framework grants GNNs the ability of non-local aggregation but lets the end-to-end training process determine whether to use non-local information . The back-propagation from the supervised loss will tune the calibration vector c and encourage ẑi to capture useful information that is not encoded by zi . In the case of disassortative graphs , ẑi usually needs to aggregate information from distant but informative nodes . Hence , the calibration vector c tends to arrange the attention-guided sorting to put distant but informative nodes together , as demonstrated experimentally in Section 4.5 . On the other hand , nodes within the local neighborhood are usually much more informative than distant nodes in assortative graphs . In this situation , ẑi may simply perform local aggregation that is similar to GNNs . In Section 4 , we demonstrate the effectiveness of our proposed non-local aggregation framework on six disassortative graph datasets . In particular , we achieve the state-of-the-art performance on all the datasets with significant improvements over previous methods .

Summary: This paper targets on addressing the node embedding problem in disassortative graphs. A non-local aggregation framework is proposed, since local aggregation may be harmful for some disassortative graphs. To address the high computational cost in the recent Geom-GCN model that has an attention-like step to compute the Euclidean distance between every pair of nodes, an idea of attention-guided sorting is introduced. It learns an ordering of nodes, such that distant but informative nodes are put near each other. The sorting order depends on the attention scores computed with the local embedding vector of a node. Then Covn(.) function is applied on the sorted sequence of local node embeddings to obtain the non-local embedding. The final node embedding is then the concatenation of the local and non-local embedding, which is used for node classification.