GRAPHS, ENTITIES, AND STEP MIXTURE

Graph neural networks have shown promising results on representing and analyzing diverse graph-structured data such as social , citation , and protein interaction networks . Existing approaches commonly suffer from the oversmoothing issue , regardless of whether policies are edge-based or node-based for neighborhood aggregation . Most methods also focus on transductive scenarios for fixed graphs , leading to poor generalization performance for unseen graphs . To address these issues , we propose a new graph neural network model that considers both edge-based neighborhood relationships and node-based entity features , i.e . Graph Entities with Step Mixture via random walk ( GESM ) . GESM employs a mixture of various steps through random walk to alleviate the oversmoothing problem and attention to use node information explicitly . These two mechanisms allow for a weighted neighborhood aggregation which considers the properties of entities and relations . With intensive experiments , we show that the proposed GESM achieves state-of-the-art or comparable performances on four benchmark graph datasets comprising transductive and inductive learning tasks . Furthermore , we empirically demonstrate the significance of considering global information . The source code will be publicly available in the near future . 1 INTRODUCTION . Graphs are universal data representations that exist in a wide variety of real-world problems , such as analyzing social networks ( Perozzi et al. , 2014 ; Jia et al. , 2017 ) , forecasting traffic flow ( Manley , 2015 ; Yu et al. , 2017 ) , and recommending products based on personal preferences ( Page et al. , 1999 ; Kim et al. , 2019 ) . Owing to breakthroughs in deep learning , recent graph neural networks ( GNNs ) ( Scarselli et al. , 2008 ) have achieved considerable success on diverse graph problems by collectively aggregating information from graph structures ( Wang et al. , 2018 ; Xu et al. , 2018 ; Gao & Ji , 2019 ) . As a result , much research in recent years has focused on how to aggregate the feature representations of neighbor nodes so that the dependence of graphs is effectively utilized . The majority of studies have predominantly depended on edges to aggregate the neighboring nodes ’ features . These edge-based methods are premised on the concept of relational inductive bias within graphs ( Battaglia et al. , 2018 ) , which implies that two connected nodes have similar properties and are more likely to share the same label ( Kipf & Welling , 2017 ) . While this approach leverages graphs ’ unique property of capturing relations , it appears less capable of generalizing to new or unseen graphs ( Wu et al. , 2019b ) . To improve the neighborhood aggregation scheme , some studies have incorporated node information ; They fully utilize node information and reduce the effects of relational ( edge ) information . A recent approach , graph attention networks ( GAT ) , employs the attention mechanism so that weights used for neighborhood aggregation differ according to the feature of nodes ( Veličković et al. , 2018 ) . This approach has yielded impressive performance and has shown promise in improving generalization for unseen graphs . Regardless of neighborhood aggregation schemes , most methods , however , suffer from a common problem where neighborhood information is considered to a limited degree ( Klicpera et al. , 2019 ) . For example , graph convolutional networks ( GCNs ) ( Kipf & Welling , 2017 ) only operate on data that are closely connected due to oversmoothing ( Li et al. , 2018 ) , which indicates the “ washing out ” of remote nodes ’ features via averaging . Consequently , information becomes localized and access to global information is restricted ( Xu et al. , 2018 ) , leading to poor performance on datasets in which only a small portion is labeled ( Li et al. , 2018 ) . In order to address the aforementioned issues , we propose a novel method , Graph Entities with Step Mixture via random walk ( GESM ) , which considers information from all nodes in the graph and can be generalized to new graphs by incorporating random walk and attention . Random walk enables our model to be applicable to previously unseen graph structures , and a mixture of random walks alleviates the oversmoothing problem , allowing global information to be included during training . Hence , our method can be effective , particularly for nodes in the periphery or a sparsely labeled dataset . The attention mechanism also advances our model by considering node information for aggregation . This enhances the generalizability of models to diverse graph structures . To validate our approach , we conducted experiments on four standard benchmark datasets : Cora , Citeseer , and Pubmed , which are citation networks for transductive learning , and protein-protein interaction ( PPI ) for inductive learning , in which test graphs remain unseen during training . In addition to these experiments , we verified whether our model uses information of remote nodes by reducing the percentage of labeled data . The experimental results demonstrate the superior performance of GESM on inductive learning as well as transductive learning for datasets . Moreover , our model achieved enhanced accuracy for datasets with reduced label rates , indicating the contribution of global information . The key contributions of our approach are as follows : • We present graphs with step mixture via random walk , which can adaptively consider local and global information , and demonstrate its effectiveness through experiments on public benchmark datasets with few labels . • We propose Graph Entities with Step Mixture via random walk ( GESM ) , an advanced model which incorporates attention , and experimentally show that it is applicable to both transductive and inductive learning tasks , for both nodes and edges are utilized for the neighborhood aggregation scheme . • We empirically demonstrate the importance of propagation steps by analyzing its effect on performance in terms of inference time and accuracy . 2 BACKGROUNDS . 2.1 RANDOM WALKS Random walk , which is a widely used method in graph theory , mathematically models how node information propagates throughout the graph . As shown in Figure 1 , random walk refers to randomly moving to neighbor nodes from the starting node in a graph . For a given graph , the transition matrix P , which describes the probabilities of transition , can be formulated as follows : P = AD−1 ( 1 ) where A denotes the adjacency matrix of the graph , and D the diagonal matrix with a degree of nodes . The probability of moving from one node to any of its neighbors is equal , and the sum of the probabilities of moving to a neighboring node adds up to one . Let ut be the distribution of the random walk at step t ( u0 represents the starting distribution ) . The t step random walk distribution is equal to multiplying P , the transition matrix , t times . In other words , u1 = Pu0 ut = Put−1 = P tu0 . ( 2 ) The entries of the transition matrix are all positive numbers , and each column sums up to one , indicating that P is a matrix form of the Markov chain with steady-state . One of the eigenvalues is equal to 1 , and its eigenvector is a steady-state ( Strang , 1993 ) . Therefore , even if the transition matrix is infinitely multiplied , convergence is guaranteed . 2.2 ATTENTION . The attention mechanism was introduced in sequence-to-sequence modeling to solve long-term dependency problems that occur in machine translation ( Bahdanau et al. , 2015 ) . The key idea of attention is allowing the model to learn and focus on what is important by examining features of the hidden layer . In the case of GNNs ( Scarselli et al. , 2008 ) , GATs ( Veličković et al. , 2018 ) achieved stateof-the-art performance by using the attention mechanism . Because the attention mechanism considers the importance of each neighboring node , node features are given more emphasis than structural information ( edges ) during the propagation process . Consequently , using attention is advantageous for training and testing graphs with different node features but the same structures ( edges ) . Given the many benefits of attention , we incorporate the attention mechanism to our model to fully utilize node information . The attention mechanism enables different importance values to be assigned to nodes of the same neighborhood , so combining attention with mixture-step random walk allows our model to adaptively highlight features with salient information in a global scope . 3 OUR PROPOSED METHODS . Let G = ( V , E ) be a graph , where V and E denote the sets of nodes and edges , respectively . Nodes are represented as a feature matrix X ∈ Rn×f , where n and f respectively denote the number of nodes and the input dimension per node . A label matrix is Y ∈ Rn×c with the number of classes c , and a learnable weight matrix is denoted by W . The adjacency matrix of graph G is represented as A ∈ Rn×n . The addition of self-loops to the adjacency matrix is Ã = A + In , and the column normalized matrix of Ã is ˆ̃A = ÃD−1 with ˆ̃A0 = In . 3.1 GRAPH STEP MIXTURE : BASE APPROACH Most graph neural networks suffer from the oversmoothing issue along with localized aggregation . Although JK-Net ( Xu et al. , 2018 ) tried to handle oversmoothing by utilizing GCN blocks with mulitple propagation , it could not completely resolve the issue as shown in Figure 4b . We therefore propose Graph Step Mixture ( GSM ) , which not only separates the node embedding and propagation process but also tackles oversmoothing and localized aggregation issues through a mixture of random walk steps . GSM has a simple structure that is composed of three stages , as shown in Figure 2 . Input X passes through a fully connected layer with a nonlinear activation . The output is then multiplied by a normalized adjacency matrix ˆ̃A for each random walk step that is to be considered . The results for each step are concatenated and fed into another fully connected layer , giving the final output . The entire propagation process of GSM can be formulated as : ZGSM = softmax ( ( sn k=0 ˆ̃Akσ ( XW0 ) ) W1 ) , ( 3 ) where f is the concatenation operation , s is the maximum number of steps considered for aggregation , and ˆ̃Ak is the normalized adjacency matrix ˆ̃A multiplied k times . As can be seen from Equation 3 , weights are shared across nodes . In our method , the adjacency matrix ˆ̃A is an asymmetric matrix , which is generated by random walks and flexible to arbitrary graphs . On the other hand , prior methods such as JK-Net ( Xu et al. , 2018 ) and MixHop ( Abu-El-Haija et al. , 2019 ) , use a symmetric Laplacian adjacency matrix , which limits graph structures to given fixed graphs . For the concatenation operation , localized sub-graphs are concatenated with global graphs , which allows the neural network to adaptively select global and local information through learning ( see Figure 3 ) . While traditional graph convolution methods consider aggregated information within three steps by A ( A ( AXW ( 0 ) ) W ( 1 ) ) W ( 2 ) , our method can take all previous aggregations into account by ( A0XW | A1XW | A2XW | A3XW ) . 3.2 GRAPH ENTITY STEP MIXTURE : GSM ENHANCED WITH ATTENTION . To develop our base model which depends on edge information , we additionally adopt the attention mechanism so that node information is emphasized for aggregation , i.e. , Graph Entity Step Mixture ( GESM ) . We simply modify the nonlinear transformation of the first fully connected layer in GSM by replacing it with the attention mechanism denoted byHmulti ( see Equations 3 and 4 ) . As described in Equation 4 , we employ multi-head attention , where Hmulti is the concatenation of m attention layers and α is the coefficient of attention computed using concatenated features of nodes and its neighboring nodes . ZGESM = softmax ( ( sn k=0 ˆ̃AkHmulti ) W1 ) , Hmulti = mn i=1 σ ( αiXW i 0 ) ( 4 ) By incorporating attention to our base model , we can avoid or ignore noisy parts of the graph , providing a guide for random walk ( Lee et al. , 2018 ) . Utilizing attention can also improve combinatorial generalization for inductive learning , where training and testing graphs are completely different . In particular , datasets with the same structure but different node information can benefit from our method because these datasets can only be distinguished by node information . Focusing on node features for aggregation can thus provide more reliable results in inductive learning . The time complexity of our base model is O ( s × l × h ) , where s is the maximum number of steps considered for aggregation , l is the number of non-zero entries in the adjacency matrix , and h is the hidden feature dimension . As suggested by Abu-El-Haija et al . ( 2019 ) , we can assume h < < l under realistic assumptions . Our model complexity is , therefore , highly efficient with time complexity O ( s× l ) , which is on par with vanilla GCN ( Kipf & Welling , 2017 ) .

Summary: This paper presents two models, namely GSM and GESM, to tackle the problem of transductive and inductive node classification. GSM is operating on asymmetric transition matrices and works by stacking propagation layers of different locality, where the final prediction is based on all propagation steps (JK concatenation style). GESM builds upon GSM and introduces a multi-headed attention layer applied on the initial feature matrix to guide the propagation layers. The models are evaluated on four common benchmark datasets and achieve state-of-the-art performance, especially when the training label rate is reduced.