Non-Local Graph Neural Networks

1 INTRODUCTION . Graph neural networks ( GNNs ) process graphs and map each node to an embedding vector ( Zhang et al. , 2018b ; Wu et al. , 2019 ) . These node embeddings can be directly used for node-level applications , such as node classification ( Kipf & Welling , 2017 ) and link prediction ( Schütt et al. , 2017 ) . In addition , they can be used to learn the graph representation vector with graph pooling ( Ying et al. , 2018 ; Zhang et al. , 2018a ; Lee et al. , 2019 ; Yuan & Ji , 2020 ) , in order to fit graph-level tasks ( Yanardag & Vishwanathan , 2015 ) . Many variants of GNNs have been proposed , such as ChebNets ( Defferrard et al. , 2016 ) , GCNs ( Kipf & Welling , 2017 ) , GraphSAGE ( Hamilton et al. , 2017 ) , GATs ( Veličković et al. , 2018 ) , LGCN ( Gao et al. , 2018 ) and GINs ( Xu et al. , 2019 ) . Their advantages have been shown on various graph datasets and tasks ( Errica et al. , 2020 ) . However , these GNNs share a multilayer local aggregation framework , which is similar to convolutional neural networks ( CNNs ) ( LeCun et al. , 1998 ) on grid-like data such as images and texts . In recent years , the importance of non-local aggregation has been demonstrated in many applications in the field of computer vision ( Wang et al. , 2018 ; 2020 ) and natural language processing ( Vaswani et al. , 2017 ) . In particular , the attention mechanism has been widely explored to achieve non-local aggregation and capture long-range dependencies from distant locations . Basically , the attention mechanism measures the similarity between every pair of locations and enables information to be communicated among distant but similar locations . In terms of graphs , non-local aggregation is also crucial for disassortative graphs , while previous studies of GNNs focus on assortative graph datasets ( Section 2.2 ) . In addition , we find that local aggregation is even harmful for some disassortative graphs ( Section 4.3 ) . The recently proposed Geom-GCN ( Pei et al. , 2020 ) explores to capture longrange dependencies in disassortative graphs . It contains an attention-like step that computes the Euclidean distance between every pair of nodes . However , this step is computationally prohibitive for large-scale graphs , as the computational complexity is quadratic in the number of nodes . In addition , Geom-GCN employs pre-trained node embeddings ( Tenenbaum et al. , 2000 ; Nickel & Kiela , 2017 ; Ribeiro et al. , 2017 ) that are not task-specific , limiting the effectiveness and flexibility . In this work , we propose a simple yet effective non-local aggregation framework for GNNs . At the heart of the framework lies an efficient attention-guided sorting , which enables non-local aggregation through classic local aggregation operators in general deep learning . The proposed framework can be flexibly used to augment common GNNs with low computational costs . Based on the framework , we build various efficient non-local GNNs . In addition , we perform detailed analysis on existing disassortative graph datasets , and apply different non-local GNNs accordingly . Experimental results show that our non-local GNNs significantly outperform previous state-of-the-art methods on node classification tasks on six benchmark datasets of disassortative graphs . 2 BACKGROUND AND RELATED WORK . 2.1 GRAPH NEURAL NETWORKS . We focus on learning the embedding vector for each node through graph neural networks ( GNNs ) . Most existing GNNs are inspired by convolutional neural networks ( CNNs ) ( LeCun et al. , 1998 ) and follow a local aggregation framework . In general , each layer of GNNs scans every node in the graph and aggregates local information from directly connected nodes , i.e. , the 1-hop neighbors . Specifically , a common layer of GNNs performs a two-step processing similar to the depthwise separable convolution ( Chollet , 2017 ) : spatial aggregation and feature transformation . The first step updates each node embedding using embedding vectors of spatially neighboring nodes . For example , GCNs ( Kipf & Welling , 2017 ) and GATs ( Veličković et al. , 2018 ) compute a weighted sum of node embeddings within the 1-hop neighborhood , where weights come from the degree of nodes and the interaction between nodes , respectively . GraphSAGE ( Hamilton et al. , 2017 ) applies the max pooling , while GINs ( Xu et al. , 2019 ) simply sums the node embeddings . The feature transformation step is similar to the 1×1 convolution , where each node embedding vector is mapped into a new feature space through a shared linear transformation ( Kipf & Welling , 2017 ; Hamilton et al. , 2017 ; Veličković et al. , 2018 ) or multilayer perceptron ( MLP ) ( Xu et al. , 2019 ) . Different from these studies , LGCN ( Gao et al. , 2018 ) explores to directly apply the regular convolution through top-k ranking . Nevertheless , each layer of these GNNs only aggregates local information within the 1-hop neighborhood . While stacking multiple layers can theoretically enable communication between nodes across the multi-hop neighborhood , the aggregation is essentially local . In addition , deep GNNs usually suffer from the over-smoothing problem ( Xu et al. , 2018 ; Li et al. , 2018 ; Chen et al. , 2020 ) . 2.2 ASSORTATIVE AND DISASSORTATIVE GRAPHS . There are many kinds of graphs in the literature , such as citation networks ( Kipf & Welling , 2017 ) , community networks ( Chen et al. , 2020 ) , co-occurrence networks ( Tang et al. , 2009 ) , and webpage linking networks ( Rozemberczki et al. , 2019 ) . We focus on graph datasets corresponding to the node classification tasks . In particular , we categorize graph datasets into assortative and disassortative ones ( Newman , 2002 ; Ribeiro et al. , 2017 ) according to the node homophily in terms of labels , i.e. , how likely nodes with the same label are near each other in the graph . Assortative graphs refer to those with a high node homophily . Common assortative graph datasets are citation networks and community networks . On the other hand , graphs in disassortative graph datasets contain more nodes that have the same label but are distant from each other . Example disassortative graph datasets are co-occurrence networks and webpage linking networks . As introduced above , most existing GNNs perform local aggregation only and achieve good performance on assortative graphs ( Kipf & Welling , 2017 ; Hamilton et al. , 2017 ; Veličković et al. , 2018 ; Gao et al. , 2018 ) . However , they may fail on disassortative graphs , where informative nodes in the same class tend to be out of the local multi-hop neighborhood and non-local aggregation is needed . Thus , in this work , we explore the non-local GNNs . 2.3 ATTENTION MECHANISM . The attention mechanism ( Vaswani et al. , 2017 ) has been widely used in GNNs ( Veličković et al. , 2018 ; Gao & Ji , 2019 ; Knyazev et al. , 2019 ) as well as other deep learning models ( Yang et al. , 2016 ; Wang et al. , 2018 ; 2020 ) . A typical attention mechanism takes three groups of vectors as inputs , namely the query vector q , key vectors ( k1 , k2 , . . . , kn ) , value vectors ( v1 , v2 , . . . , vn ) . Note that key and value vectors have a one-to-one correspondence and can be the same sometimes . The attention mechanism computes the output vector o as ai = ATTEND ( q , ki ) ∈ R , i = 1 , 2 , . . . , n ; o = ∑ i aivi , ( 1 ) where the ATTEND ( · ) function could be any function that outputs a scalar attention score ai from the interaction between q and ki , such as dot product ( Gao & Ji , 2019 ) or even a neural net- work ( Veličković et al. , 2018 ) . The definition of the three groups of input vectors depends on the models and applications . Notably , existing GNNs usually use the attention mechanism for local aggregation ( Veličković et al. , 2018 ; Gao & Ji , 2019 ) . Specifically , when aggregating information for node v , the query vector is the embedding vector of v while the key and value vectors come from node embeddings of v ’ s directly connected nodes . And the process is iterated for each v ∈ V . It is worth noting that the attention mechanism can be easily extended for non-local aggregation ( Wang et al. , 2018 ; 2020 ) , by letting the key and value vectors correspond to all the nodes in the graph when aggregating information for each node . However , it is computationally prohibitive given large-scale graphs , as iterating it for each node in a graph of n nodes requires O ( n2 ) time . In this work , we propose a novel non-local aggregation method that only requires O ( n log n ) time . 3 THE PROPOSED METHOD . 3.1 NON-LOCAL AGGREGATION WITH ATTENTION-GUIDED SORTING . We consider a graph G = ( V , E ) , where V is the set of nodes and E is the set of edges . Each edge e ∈ E connects two nodes so that E ⊆ V ×V . Each node v ∈ V has a node feature vector xv ∈ Rd . The k-hop neighborhood of v refers to the set of nodes Nk ( v ) that can reach v within k edges . For example , the set of v ’ s directly connected nodes is its 1-hop neighborhood N1 ( v ) . Our proposed non-local aggregation framework is composed of three steps , namely local embedding , attention-guided sorting , and non-local aggregation . In the following , we describe them one by one . Local Embedding : Our proposed framework is built upon a local embedding step that extracts local node embeddings from the node feature vectors . The local embedding step can be as simple as zv = MLP ( xv ) ∈ Rf , ∀v ∈ V. ( 2 ) The MLP ( · ) function is a multilayer perceptron ( MLP ) , and f is the dimension of the local node embedding zv . Note that the MLP ( · ) function is shared across all the nodes in the graph . Applying MLP only takes the node itself into consideration without aggregating information from the neighborhood . This property is very important on some disassortative graphs , as shown in Section 4.3 . On the other hand , graph neural networks ( GNNs ) can be used as the local embedding step as well , so that our proposed framework can be easily employed to augment existing GNNs . As introduced in Section 2.1 , modern GNNs perform multilayer local aggregation . Typically , for each node , one layer of a GNN aggregates information from its 1-hop neighborhood . Stacking L such local aggregation layers allows each node to access information that is L hops away . To be specific , the ` -th layer of a L-layer GNN ( ` = 1 , 2 , . . . , L ) can be described as z ( ` ) v = TRANSFORM ( ` ) ( AGGREGATE ( ` ) ( { z ( ` −1 ) u : u ∈ N1 ( v ) ∪ v } ) ) ∈ Rf , ∀v ∈ V , ( 3 ) where z ( 0 ) v = xv , and zv = z ( L ) v represents the local node embedding . The AGGREGATE ( ` ) ( · ) and TRANSFORM ( ` ) ( · ) functions represent the spatial aggregation and feature transformation step introduced in Section 2.1 , respectively . With the above framework , GNNs can capture the node feature information from nodes within a local neighborhood as well as the structural information . When either MLP or GNNs is used as the local embedding step , the local node embedding zv only contains local information of a node v. However , zv can be used to guide non-local aggregation , as distant but informative nodes are likely to have similar node features and local structures . Based on this intuition , we propose the attention-guided sorting to enable the non-local aggregation . Attention-Guided Sorting : The basic idea of the attention-guided sorting is to learn an ordering of nodes , where distant but informative nodes are put near each other . Specifically , given the local node embedding zv obtained through the local embedding step , we compute one set of attention scores by av = ATTEND ( c , zv ) ∈ R , ∀v ∈ V , ( 4 ) where c is a calibration vector that is randomly initialized and jointly learned during training ( Yang et al. , 2016 ) . In this attention operator , c serves as the query vector and zv are the key vectors . In addition , we also treat zv as the value vectors . However , unlike the attention mechanism introduced in Section 2.3 , we use the attention scores to sort the value vectors instead of computing a weighted sum to aggregating them . Note that originally there is no ordering among nodes in a graph . To be specific , as av and zv have one-to-one correspondence through Equation ( 4 ) , sorting the attention scores in non-decreasing order into ( a1 , a2 , . . . , an ) provides an ordering among nodes , where n = |V | is the number of nodes in the graph . The resulting sequence of local node embeddings can be denoted as ( z1 , z2 , . . . , zn ) . The attention process in Equation ( 4 ) can be also understood as a projection of local node embeddings onto a 1-dimensional space . The projection depends on the concrete ATTEND ( · ) function and the calibration vector c. As indicated by its name , the calibration vector c is used to calibrate the 1-dimensional space , in order to push distant but informative nodes close to each other in this space . This goal is fulfilled through the following non-local aggregation step and the training of the calibration vector c , as demonstrated below . Non-Local Aggregation : We point out that , with the attention-guided sorting , the non-local aggregation can be achieved by convolution , the most common local aggregation operator in deep learning . Specifically , given the sorted sequence of local node embeddings ( z1 , z2 , . . . , zn ) , we compute ( ẑ1 , ẑ2 , . . . , ẑn ) = CONV ( z1 , z2 , . . . , zn ) , ( 5 ) where the CONV ( · ) function represents a 1D convolution with appropriate padding . Note that the CONV ( · ) function can be replaced by a 1D convolutional neural network as long as the number of input and output vectors remains the same . To see how the CONV ( · ) function performs non-local aggregation with the attention-guided sorting , we take an example where the CONV ( · ) function is a 1D convolution of kernel size 2s + 1 . In this case , ẑi is computed from ( zi+s , . . . , zi−s ) , corresponding to the receptive field of the CONV ( · ) function . As a result , if the attention-guided sorting leads to ( zi+s , . . . , zi−s ) containing nodes that are distant but informative to zi , the output ẑi aggregates non-local information . Another view is that we can consider the attention-guided sorting as re-connects nodes in the graph , where ( zi+s , . . . , zi−s ) can be treated as the 1-hop neighborhood of zi . After the CONV ( · ) function , ẑi and zi are concatenated as the input to a classifier to predict the label of the corresponding node , where both non-local and local dependencies can be captured . In order to enable the end-to-end training of the calibration vector c , we modify Equation ( 5 ) into ( ẑ1 , ẑ2 , . . . , ẑn ) = CONV ( a1z1 , a2z2 , . . . , anzn ) , ( 6 ) where we multiply the attention score with the corresponding local node embedding . As a result , the calibration vector c receives gradients through the attention scores during training . The remaining question is how to make sure that the attention-guided sorting pushes distant but informative nodes together . The short answer is that it is not necessary to guarantee this , as the requirement of non-local aggregation depends on the concrete graphs . In fact , our proposed framework grants GNNs the ability of non-local aggregation but lets the end-to-end training process determine whether to use non-local information . The back-propagation from the supervised loss will tune the calibration vector c and encourage ẑi to capture useful information that is not encoded by zi . In the case of disassortative graphs , ẑi usually needs to aggregate information from distant but informative nodes . Hence , the calibration vector c tends to arrange the attention-guided sorting to put distant but informative nodes together , as demonstrated experimentally in Section 4.5 . On the other hand , nodes within the local neighborhood are usually much more informative than distant nodes in assortative graphs . In this situation , ẑi may simply perform local aggregation that is similar to GNNs . In Section 4 , we demonstrate the effectiveness of our proposed non-local aggregation framework on six disassortative graph datasets . In particular , we achieve the state-of-the-art performance on all the datasets with significant improvements over previous methods .

Summary: The goal of the paper is to perform node classification for graphs. The authors propose a strategy to augment message passing graph neural networks with information from non-local nodes in the graph - with a focus on dis-assortative graphs. Dis-assortative graphs are graph datasets - where nodes with identical node labels are distant from each other in terms of edge connectivity.