Teleport Graph Convolutional Networks

1 INTRODUCTION . Graph neural networks ( GNNs ) have shown great capability in solving challenging tasks on graph data such as node classification ( Grover & Leskovec , 2016 ; Kipf & Welling , 2017 ; Veličković et al. , 2017 ; Gao et al. , 2018 ) , graph classification ( Xu et al. , 2018 ; Gao & Ji , 2019 ; You et al. , 2019 ) , and link prediction ( Zhang & Chen , 2018 ; Chen et al. , 2019 ; Zhou et al. , 2019 ) . Most graph convolutional networks are based on message-passing operations , in which each node aggregates information from its neighboring nodes . To enable a larger receptive field ( Chen et al. , 2016 ) , GNNs need to stack multiple layers , which is straightforward but can result in several issues . Firstly , stacking multiple layers involves massive trainable parameters , which consequently increases the risk of over-fitting . Secondly , message-passing operations mostly use averaging to combine the aggregated features , which significantly reduces the distinguishability of network embeddings . From this point , GNNs that are based on message-passing operations can not use deep network architecture due to these limitations . Some works such as Geom-GCN ( Pei et al. , 2020 ) try to solve these issues by involving more nodes in the feature aggregation process . However , Geom-GCN doesn ’ t consider the original graph topology information when generating the additional set of nodes for aggregation , which can neglect some relevant nodes from a structural perspective . To address the above limitations and increase the receptive field effectively , we propose a teleport graph convolution layer ( TeleGCL ) that uses teleport functions to select highly-relevant nodes at the global scope . A teleport function computes relevances between the center node and other nodes beyond the local neighborhood . The nodes with particular relevances are teleported for the center node . Here , the selection of teleported nodes is not restricted by the graph topology . This enables the center node to gather information from a larger neighborhood without going deep , which helps to avoid over-fitting and over-smoothing issues . In particular , we propose two teleport functions ; those are structure-aware and feature-aware teleport functions . They compute the nodes ’ relevances from graph structural perspective and node features perspective , respectively . Based on our TeleGCL , we build a family of teleport graph convolutional networks . The empirical results on graph and node classification tasks demonstrate the effectiveness of our proposed methods . 2 BACKGROUND AND RELATED WORK . In this section , we describe message-passing operations on graph data and geometric graph convolutional networks . Graph neural networks ( Fan et al. , 2019 ; Wu et al. , 2019 ; Morris et al. , 2019 ; Wu et al. , 2020 ) have achieved state-of-the-art performances on various challenging tasks in the field of network embedding . The mainstream of graph deep learning operations follows a message-passing schema . In a message-passing operation , each node sends its features , known as message , to its neighboring nodes in the graph . Then each node aggregates messages from its neighborhood and uses them to update its features . When combing the aggregated features , different strategies can be applied . In the graph convolution layer ( GCN ) ( Kipf & Welling , 2017 ) , features from neighboring nodes are given equal weights in the aggregation process . To assign different weights to different neighboring nodes , the graph attention network ( Veličković et al. , 2017 ) employs an attention mechanism to compute aggregation weights . Based on these message-passing operations , graph neural networks stack multiple layers , which enables a larger receptive field . Recently , some research works try to perform message passing beyond the local neighborhood . Pei et al . ( 2020 ) proposed to construct a continuous latent space that enables graph neural networks to perform feature learning in the latent space . To be specific , it first projects nodes ’ features to a 2-dimensional latent and continuous space . Based on the latent space , a structural neighborhood is constructed based on the Euclidean distance of each pair of nodes in the 2-dimensional space . In this process , the construction of structural features does not consider the graph connectivity information in the graph . Thus , the structural neighborhood in ( Pei et al. , 2020 ) is still built on node features without considering the graph topology . In this work , we propose a method to generate structure-aware features for each node . In particular , we use the graph connectivity and similarity information with the neighboring nodes and construct a feature vector for each node . By considering graph connectivity , our constructed structural features can reflect graph topology information . 3 TELEPORT GRAPH CONVOLUTIONAL NETWORKS . In this work , we propose the teleport graph convolution layer ( TeleGCL ) that enables a center node to aggregate information beyond regular neighborhood structure by using some teleport functions . To enable effective node teleportation , we propose two teleport functions from structure-aware and feature-aware perspectives . Specifically , we propose a novel method to construct structural features for nodes , which can be used by structure-aware functions to select relevant nodes . Based on our TeleGCL , we propose the teleport graph convolutional networks for network embedding learning . 3.1 LIMITATIONS OF MESSAGE-PASSING OPERATIONS . Currently , most graph convolution networks are based on message-passing operations . In a messagepassing operation , each node aggregates information from its neighboring nodes that usually are the one-hop neighborhood . Intuitively , it is beneficial to use information from a large neighborhood for network embedding learning . To enlarge the receptive field , a straight way is to stack multiple message-passing layers . A graph convolutional network with k layers enables nodes to receive information from a k-hop neighborhood . However , this method results in two issues . Firstly , it increases the risk of over-fitting by involving much more trainable parameters . The number of trainable parameters in the network increases when stacking multiple layers . Unlike regular convolutional neural networks , there is no effective graph pooling layer that can enlarge the receptive field without involving trainable parameters . Stacking many graph convolution layers will inevitably increase the risk of over-fitting . Secondly , stacking multiple layers will reduce the distinguishability of network embeddings , which is often referred to as the over-smoothing issue ( Pei et al. , 2020 ) . Due to the invariant property in graph structures , message-passing operations can not learn trainable weights in the aggregation process ( Kipf & Welling , 2017 ; Gao et al. , 2018 ) . Averaging operation is usually used for information aggregation from the neighborhood . Consequently , information from relevant distant nodes will be diluted and each node carries similar information . In this work , we propose a teleport graph convolution layer to address this issue . This layer enables each node to aggregate information from a set of relevant nodes that are not directly connected to the center node in the original graph structure . Teleport functions are used to determine the relevant nodes from different perspectives . 3.2 TELEPORT GRAPH CONVOLUTION LAYER . To address the limitations in message-passing operations , we propose the teleport graph convolution layer ( TeleGCL ) , which enables nodes to aggregate information beyond their local neighborhoods . In this layer , we employ multiple teleport functions to generate neighborhoods for each node . The teleport functions select some nodes that are relevant but not directly connected . Since these nodes are teleported from a global context , the receptive field of each node can be effectively enlarged . We require these functions to be permutation invariant such that the property retains in this layer . Given a graph G = ( V , E ) , where n = |V | , each node v ∈ V is associated with a feature vector xv ∈ Rd , and each edge ( u , v ) ∈ E connects node u and node v in the graph . X = [ x1 , x2 , . . . , xn ] ∈ Rd×n and A ∈ Rn×n are the feature matrix and adjacency matrix , respectively . A teleport function g ( v , G ) → N takes as input a node v and outputs a neighborhood N that includes a set of relevant nodes for node v ’ s feature aggregation . In Section 3.3 and Section 3.4 , we propose two teleport functions that can construct structure-aware and feature-aware neighborhoods . Suppose we have m teleport functions , node v aggregates information from a neighborhood N ( v ) = { Nl ( v ) , N1 ( v ) , N2 ( v ) , . . . , Nm ( v ) } , where Nl ( v ) = { u|u ∈ V , ( v , u ) ∈ E } is the local neighborhood and Ni ( v ) = gi ( v , G ) is a neighborhood created by the ith teleport function . Based on this neighborhood , the layer-wise propagation of TeleGCL ` for node v is formulated as x ( ` ) v = σ 1 |N ( v ) | ∑ u∈N ( v ) x ( ` −1 ) u , ( 1 ) where σ denotes an activation function such as ReLU ( Nair & Hinton , 2010 ) . By using teleport functions , a node can aggregate features beyond the local neighborhood , thereby leading to a larger receptive field . In Section 3.2 , we propose the teleport graph convolution layer that enables feature aggregation regardless of the original graph structure . A TeleGCL highly depends on teleport functions which select distant nodes in the feature aggregation process . In previous works ( Pei et al. , 2020 ) , teleport functions are mainly based on node features while neglecting the graph structural information . The graph topology information should be considered to include nodes that share the same structural patterns such as graph motifs ( Ren & Jin , 2020 ) . In this section , we propose two teleport functions ; those are structure-aware teleport function and feature-aware teleport function . They select teleport nodes based on graph topology and node features , respectively . 3.3 STRUCTURE-AWARE TELEPORT FUNCTION . Structure-aware teleport function focuses on selecting nodes based on the graph topology . In a graph , the nodes that share the same structural pattern contain related features . It is desirable for a node to aggregate information from these relevant nodes . A structure-aware function can be used to capture relevant nodes from a graph structural perspective . With a structure-aware function , the teleported nodes for node v are selected as : Ns ( v ) = { u|u ∈ V , ( v , u ) /∈ E , ts ( v , u ) > θ1 } , ( 2 ) where ts ( v , u ) is a structure-aware function that computes the relevance of two nodes from a structural perspective . Here , θ1 is used to determine if node u is relevant . In this work , we propose a novel structure-aware teleport function , which computes the relevance of two nodes by checking if they share the same structural pattern . Our proposed method is based on an intuition that the nodes with the same structural pattern have similar connections with their neighboring nodes . From this point , we create a structure feature vector for each node , which can reflect its structural pattern such as graph motifs . For each node , we first compute its similarity scores with its neighboring nodes . Then we rank these similarity scores and use them as the structural feature of this node . To be specific , the structural feature vector yv for node v is constructed as wv = X Txv , ∈Rn ( 3 ) idxv = rankk ( wv ◦A : ,v ) , ∈Rk ( 4 ) yv = wv ( idxv ) , ∈Rk ( 5 ) where ◦ denotes an element-wise vector multiplication , and A : ,v is the vth column of the adjacency matrix . rankk operator ranks the similarity scores and outputs the indices of the top-k values in wv . wv ( idxv ) returns a subset of rows in wv indexed by idxv . We first compute the similarity scores between node v and its neighboring nodes in Eq . ( 3 ) . Each element wu , v in wv measures the similarity between node u and node v. In Eq . ( 4 ) , we rank these similarity scores and select the k-largest values in wv . The indices of the selected values are stored in idxv . Using indices idxv , we extract a structural feature vector yv from wv . By repeating these operations on each node , we can obtain a structural feature matrix Y = [ y1 , y2 , . . . , yn ] ∈ Rk×n for all nodes in the graph . In this way , the structural feature vector is constructed from similarity scores between the center node and its neighboring nodes . These similarity scores encode its connectivity pattern with surrounding nodes , thereby reflecting the structural information in the graph . Based on structural features , we use dot product to compute the relevance of node u and node v : ts ( u , v ) = softmax ( yTu yv ) , which can measure relevance from the perspectives of both angle and magnitude in an efficient way . As illustrated in Eq . ( 2 ) , the teleport nodes can be selected based on our constructed structural features .

Summary: A new architecture for graph neural networks, which the authors name as Teleport Graph Convolutional Networks (TGL), is proposed in this paper.  Teleport graph convolution layer is proposed to address the limitations in message-passing operations of graph neural networks:  1.  over-smoothing and 2. over-fitting. The architecture enables nodes to aggregate information beyond their local neighborhoods. TGL operates as follows: 1) it first aggregates neighbors as in normal graph neural networks, 2) it selects some nodes that are relevant but not directly connected using 2a) feature similarity and 2b) similarity of structure.