Teleport Graph Convolutional Networks

1 INTRODUCTION . Graph neural networks ( GNNs ) have shown great capability in solving challenging tasks on graph data such as node classification ( Grover & Leskovec , 2016 ; Kipf & Welling , 2017 ; Veličković et al. , 2017 ; Gao et al. , 2018 ) , graph classification ( Xu et al. , 2018 ; Gao & Ji , 2019 ; You et al. , 2019 ) , and link prediction ( Zhang & Chen , 2018 ; Chen et al. , 2019 ; Zhou et al. , 2019 ) . Most graph convolutional networks are based on message-passing operations , in which each node aggregates information from its neighboring nodes . To enable a larger receptive field ( Chen et al. , 2016 ) , GNNs need to stack multiple layers , which is straightforward but can result in several issues . Firstly , stacking multiple layers involves massive trainable parameters , which consequently increases the risk of over-fitting . Secondly , message-passing operations mostly use averaging to combine the aggregated features , which significantly reduces the distinguishability of network embeddings . From this point , GNNs that are based on message-passing operations can not use deep network architecture due to these limitations . Some works such as Geom-GCN ( Pei et al. , 2020 ) try to solve these issues by involving more nodes in the feature aggregation process . However , Geom-GCN doesn ’ t consider the original graph topology information when generating the additional set of nodes for aggregation , which can neglect some relevant nodes from a structural perspective . To address the above limitations and increase the receptive field effectively , we propose a teleport graph convolution layer ( TeleGCL ) that uses teleport functions to select highly-relevant nodes at the global scope . A teleport function computes relevances between the center node and other nodes beyond the local neighborhood . The nodes with particular relevances are teleported for the center node . Here , the selection of teleported nodes is not restricted by the graph topology . This enables the center node to gather information from a larger neighborhood without going deep , which helps to avoid over-fitting and over-smoothing issues . In particular , we propose two teleport functions ; those are structure-aware and feature-aware teleport functions . They compute the nodes ’ relevances from graph structural perspective and node features perspective , respectively . Based on our TeleGCL , we build a family of teleport graph convolutional networks . The empirical results on graph and node classification tasks demonstrate the effectiveness of our proposed methods . 2 BACKGROUND AND RELATED WORK . In this section , we describe message-passing operations on graph data and geometric graph convolutional networks . Graph neural networks ( Fan et al. , 2019 ; Wu et al. , 2019 ; Morris et al. , 2019 ; Wu et al. , 2020 ) have achieved state-of-the-art performances on various challenging tasks in the field of network embedding . The mainstream of graph deep learning operations follows a message-passing schema . In a message-passing operation , each node sends its features , known as message , to its neighboring nodes in the graph . Then each node aggregates messages from its neighborhood and uses them to update its features . When combing the aggregated features , different strategies can be applied . In the graph convolution layer ( GCN ) ( Kipf & Welling , 2017 ) , features from neighboring nodes are given equal weights in the aggregation process . To assign different weights to different neighboring nodes , the graph attention network ( Veličković et al. , 2017 ) employs an attention mechanism to compute aggregation weights . Based on these message-passing operations , graph neural networks stack multiple layers , which enables a larger receptive field . Recently , some research works try to perform message passing beyond the local neighborhood . Pei et al . ( 2020 ) proposed to construct a continuous latent space that enables graph neural networks to perform feature learning in the latent space . To be specific , it first projects nodes ’ features to a 2-dimensional latent and continuous space . Based on the latent space , a structural neighborhood is constructed based on the Euclidean distance of each pair of nodes in the 2-dimensional space . In this process , the construction of structural features does not consider the graph connectivity information in the graph . Thus , the structural neighborhood in ( Pei et al. , 2020 ) is still built on node features without considering the graph topology . In this work , we propose a method to generate structure-aware features for each node . In particular , we use the graph connectivity and similarity information with the neighboring nodes and construct a feature vector for each node . By considering graph connectivity , our constructed structural features can reflect graph topology information . 3 TELEPORT GRAPH CONVOLUTIONAL NETWORKS . In this work , we propose the teleport graph convolution layer ( TeleGCL ) that enables a center node to aggregate information beyond regular neighborhood structure by using some teleport functions . To enable effective node teleportation , we propose two teleport functions from structure-aware and feature-aware perspectives . Specifically , we propose a novel method to construct structural features for nodes , which can be used by structure-aware functions to select relevant nodes . Based on our TeleGCL , we propose the teleport graph convolutional networks for network embedding learning . 3.1 LIMITATIONS OF MESSAGE-PASSING OPERATIONS . Currently , most graph convolution networks are based on message-passing operations . In a messagepassing operation , each node aggregates information from its neighboring nodes that usually are the one-hop neighborhood . Intuitively , it is beneficial to use information from a large neighborhood for network embedding learning . To enlarge the receptive field , a straight way is to stack multiple message-passing layers . A graph convolutional network with k layers enables nodes to receive information from a k-hop neighborhood . However , this method results in two issues . Firstly , it increases the risk of over-fitting by involving much more trainable parameters . The number of trainable parameters in the network increases when stacking multiple layers . Unlike regular convolutional neural networks , there is no effective graph pooling layer that can enlarge the receptive field without involving trainable parameters . Stacking many graph convolution layers will inevitably increase the risk of over-fitting . Secondly , stacking multiple layers will reduce the distinguishability of network embeddings , which is often referred to as the over-smoothing issue ( Pei et al. , 2020 ) . Due to the invariant property in graph structures , message-passing operations can not learn trainable weights in the aggregation process ( Kipf & Welling , 2017 ; Gao et al. , 2018 ) . Averaging operation is usually used for information aggregation from the neighborhood . Consequently , information from relevant distant nodes will be diluted and each node carries similar information . In this work , we propose a teleport graph convolution layer to address this issue . This layer enables each node to aggregate information from a set of relevant nodes that are not directly connected to the center node in the original graph structure . Teleport functions are used to determine the relevant nodes from different perspectives . 3.2 TELEPORT GRAPH CONVOLUTION LAYER . To address the limitations in message-passing operations , we propose the teleport graph convolution layer ( TeleGCL ) , which enables nodes to aggregate information beyond their local neighborhoods . In this layer , we employ multiple teleport functions to generate neighborhoods for each node . The teleport functions select some nodes that are relevant but not directly connected . Since these nodes are teleported from a global context , the receptive field of each node can be effectively enlarged . We require these functions to be permutation invariant such that the property retains in this layer . Given a graph G = ( V , E ) , where n = |V | , each node v ∈ V is associated with a feature vector xv ∈ Rd , and each edge ( u , v ) ∈ E connects node u and node v in the graph . X = [ x1 , x2 , . . . , xn ] ∈ Rd×n and A ∈ Rn×n are the feature matrix and adjacency matrix , respectively . A teleport function g ( v , G ) → N takes as input a node v and outputs a neighborhood N that includes a set of relevant nodes for node v ’ s feature aggregation . In Section 3.3 and Section 3.4 , we propose two teleport functions that can construct structure-aware and feature-aware neighborhoods . Suppose we have m teleport functions , node v aggregates information from a neighborhood N ( v ) = { Nl ( v ) , N1 ( v ) , N2 ( v ) , . . . , Nm ( v ) } , where Nl ( v ) = { u|u ∈ V , ( v , u ) ∈ E } is the local neighborhood and Ni ( v ) = gi ( v , G ) is a neighborhood created by the ith teleport function . Based on this neighborhood , the layer-wise propagation of TeleGCL ` for node v is formulated as x ( ` ) v = σ 1 |N ( v ) | ∑ u∈N ( v ) x ( ` −1 ) u , ( 1 ) where σ denotes an activation function such as ReLU ( Nair & Hinton , 2010 ) . By using teleport functions , a node can aggregate features beyond the local neighborhood , thereby leading to a larger receptive field . In Section 3.2 , we propose the teleport graph convolution layer that enables feature aggregation regardless of the original graph structure . A TeleGCL highly depends on teleport functions which select distant nodes in the feature aggregation process . In previous works ( Pei et al. , 2020 ) , teleport functions are mainly based on node features while neglecting the graph structural information . The graph topology information should be considered to include nodes that share the same structural patterns such as graph motifs ( Ren & Jin , 2020 ) . In this section , we propose two teleport functions ; those are structure-aware teleport function and feature-aware teleport function . They select teleport nodes based on graph topology and node features , respectively . 3.3 STRUCTURE-AWARE TELEPORT FUNCTION . Structure-aware teleport function focuses on selecting nodes based on the graph topology . In a graph , the nodes that share the same structural pattern contain related features . It is desirable for a node to aggregate information from these relevant nodes . A structure-aware function can be used to capture relevant nodes from a graph structural perspective . With a structure-aware function , the teleported nodes for node v are selected as : Ns ( v ) = { u|u ∈ V , ( v , u ) /∈ E , ts ( v , u ) > θ1 } , ( 2 ) where ts ( v , u ) is a structure-aware function that computes the relevance of two nodes from a structural perspective . Here , θ1 is used to determine if node u is relevant . In this work , we propose a novel structure-aware teleport function , which computes the relevance of two nodes by checking if they share the same structural pattern . Our proposed method is based on an intuition that the nodes with the same structural pattern have similar connections with their neighboring nodes . From this point , we create a structure feature vector for each node , which can reflect its structural pattern such as graph motifs . For each node , we first compute its similarity scores with its neighboring nodes . Then we rank these similarity scores and use them as the structural feature of this node . To be specific , the structural feature vector yv for node v is constructed as wv = X Txv , ∈Rn ( 3 ) idxv = rankk ( wv ◦A : ,v ) , ∈Rk ( 4 ) yv = wv ( idxv ) , ∈Rk ( 5 ) where ◦ denotes an element-wise vector multiplication , and A : ,v is the vth column of the adjacency matrix . rankk operator ranks the similarity scores and outputs the indices of the top-k values in wv . wv ( idxv ) returns a subset of rows in wv indexed by idxv . We first compute the similarity scores between node v and its neighboring nodes in Eq . ( 3 ) . Each element wu , v in wv measures the similarity between node u and node v. In Eq . ( 4 ) , we rank these similarity scores and select the k-largest values in wv . The indices of the selected values are stored in idxv . Using indices idxv , we extract a structural feature vector yv from wv . By repeating these operations on each node , we can obtain a structural feature matrix Y = [ y1 , y2 , . . . , yn ] ∈ Rk×n for all nodes in the graph . In this way , the structural feature vector is constructed from similarity scores between the center node and its neighboring nodes . These similarity scores encode its connectivity pattern with surrounding nodes , thereby reflecting the structural information in the graph . Based on structural features , we use dot product to compute the relevance of node u and node v : ts ( u , v ) = softmax ( yTu yv ) , which can measure relevance from the perspectives of both angle and magnitude in an efficient way . As illustrated in Eq . ( 2 ) , the teleport nodes can be selected based on our constructed structural features .

Summary: This paper analyzed the key issues of the existing message-passing graph convolutional networks. That is, the multiply stacked layer might be over-fitting and over-smoothing. Thus it proposed to choose the neighbors from the entire graph based on the structure-aware and feature-aware relatedness rather than simply choosing the local neighborhood. However, the motivations of the proposed structure-aware and feature-aware teleport functions are not very convincing, and Table 3 shows that the performance improvement of TeleGCN might largely be induced by model architecture rather than the proposed TeleGCL.