DEEP GRAPH TREE NETWORKS

1 INTRODUCTION . Graph Neural Networks ( GNNs ) , a class of neural networks for learning on graph structured data , have been successfully applied in many areas to solve real world problems , such as link predictions in social networks ( Fan et al. , 2019 ) , pattern recognition ( Ju et al. , 2020 ; Shi & Rajkumar , 2020 ) , product recommendation and personalized search in E-commerce ( Zhu et al. , 2019 ) , fraud detection ( Wang et al. , 2019a ; Dou et al. , 2020 ) , protein interface predictions ( Fout et al. , 2017 ) , power estimation and tier design in the semiconductor industry ( Zhang et al. , 2020 ; Lu et al. , 2020 ) , traffic forecasting ( Yu et al. , 2017 ) , and natural language processing ( Yao et al. , 2019 ; Vashishth et al. , 2020 ; Wu et al. , 2021 ) . Among many different graph learning approaches , the class of spatial graph convolution based models , which adopts a message passing scheme to update node features , has gained particular attention due to its simplicity yet good performance . The most representative work among this class is the Graph Convolutional Network ( GCN ) ( Kipf & Welling , 2017 ) which limits the ChebNet ( Defferrard et al. , 2016 ) up to the 1st order polynomial , and together with some further approximation leading to a direct neighbor aggregation . The success of the GCN model ( Ying et al. , 2018 ; Hu et al. , 2020b ) has led to the rapid development in spatial convolution based models , such as the Graph Attention Network ( GAT ) ( Velickovic et al. , 2018 ) , GraphSage ( Hamilton et al. , 2017 ) , APPNP ( Klicpera et al. , 2018 ) , DAGNN ( Liu et al. , 2020 ) , and etc . Today the vanilla GCN and GAT are the two most popular baseline models . One GCN or GAT layer aggregates only the direct neighbor nodes . Although they work well in many test cases , their performance degrades as stacking multiple propagation layers to achieve larger receptive fields . Xu et al . ( 2018 ) and Li et al . ( 2018 ) attribute such degradation in performance to over-smoothing effect that nodes from different classes become indistinguishable , while Liu et al . ( 2020 ) attributes that to the intertwined propagation and transformation . A small neighborhood may not provide enough information especially when nodes are sparsely labeled ( Klicpera et al. , 2018 ; Liu et al. , 2020 ) . Many recent work has been devoted to extending the size of neighborhood utilized in graph learning . APPNP ( Klicpera et al. , 2018 ) and DAGNN ( Liu et al. , 2020 ) are two most recent deep graph models with state-of-the-art performance on several popular benchmark datasets . APPNP ( Klicpera et al. , 2018 ) is developed based on personalized PageRank to preserve local information which requires fine tuning of a teleport probability . DAGNN ( Liu et al. , 2020 ) aggregates neighbors from different hops in parallel ( in one layer ) to utilize information from a larger receptive field as compared to GCN . Although DAGNN has achieved state-of-the-art performance , it may only be applicable to adopt simple aggregation function as the one used in GCN . It could be computationally challenging to combine such multi-hop aggregation scheme with attention-based GNN models ( such as GAT , and graph transformer ( Dwivedi & Bresson , 2020 ; Hu et al. , 2020c ) ) ; because this aggregation scheme requires computation of the attention weight for each pair of a node and its k-hop neighbor , which could be a too large number . In the aggregation of k-hop neighborhood , the number of attention weights is the total number of non-zero entries in the kth power of the graph adjacency matrix A . Although the adjacency matrix is usually sparse , its kth power could be non-sparse especially for undirected graphs . For example , in the Cora dataset ( Wang et al. , 2019b ) A5 has ∼2.2 million non-zero entries while A has only ∼10k non-zero entries . Furthermore , aggregating multi-hop neighborhood in one step neglects path information which is usually important for heterogeneous graphs with rich metapath information ( Fu et al. , 2020 ; Wang et al. , 2019c ) . We propose Graph Tree Networks ( GTreeNets ) , a self-interpretive deep graph neural network architecture which originates from the tree representation of graphs . In the tree representation , each node forms its own tree where the node itself is the root node and all its neighbors up to k-hop are the subnodes . Under the tree representation , message propagates upward from the leaf nodes to the root node naturally to update the root node ’ s hidden features . This message passing scheme , which has better interpretability , is essentially different from that in the vanilla Graph Convolution Network ( GCN ) , Graph Attention Network ( GAT ) and many of their derivatives . Two scalable graph learning models are proposed within this GTreeNet architecture - Graph Tree Convolution Network ( GTCN ) and Graph Tree Attention Network ( GTAN ) , with experimentally demonstrated state-ofthe-art performance on several benchmark datasets and the capability of going deep by stacking multiple propagation layers . Our models do not require finely-tuned hyperparameter in the propagation scheme . This work is structured as follows . Section 2 illustrates the architecture of our Graph Tree Networks ( GTreeNets ) and two derivative graph learning models - Graph Tree Convolution Network ( GTCN ) and Graph Tree Attention Network ( GTAN ) , along with the analysis of model complexity . Section 3 presents the experimental results including three parts : Section 3.1 shows the performance test of our GTCN and GTAN models as compared to several popular and state-of-the-art GNN models on five benchmark datasets . Section 3.2 shows the performance test for our GTCN and GTAN models at different depths as compared to the vanilla GCN and GAT models . Section 3.3 discusses the effect of adding transformation in propagation layers in our GTCN and GTAN models . We conclude our work in Section 4 and discuss insights for future work . 2 OUR APPROACH : GRAPH TREE NETWORKS . We first introduce notations used throughout this paper . We follow the general convention to use bold uppercase and lowercase to represent matrices and vectors , respectively . The topology of a graph G = ( V , E ) with nodes V and edges E can be fully described by its adjacency matrix A and degree matrix D. |V | = N and |E| = M are the number of nodes and edges , respectively . Nu denotes the set of direct neighbors ( 1-hop neighbors ) of node u. X ∈ RN×D represents the feature map for all nodes , where each row xu ∈ RD represents the feature vector of node u with dimension of D. Y ∈ RN×C represents the class matrix for all nodes , where each row yu represents the class vector of node u with C classes . 2.1 TREE REPRESENTATION AND NETWORK STRUCTURE . Tree is a straightforward representation for the graph topology , where each node and its neighborhood form one tree with the node itself being the root node and its neighbors being the subnodes . Figure 1 illustrates the tree representation for node 1 with up to 3-hop neighbors in the sample graph . We use a directed graph just for illustration , the tree representation also works for undirected graphs . In Figure 1 , hku ∈ RF is the vector of node u ’ s output hidden features at k-hop away from the root node 1 . The final output of the root node 1 is denoted as h01 . Messages propagate upward from subnodes to the root node hop by hop in the tree . Nodes may occur multiple times at the same level in the tree ( like node 5 occurring twice at 2-hop as shown in Figure 1 ) , each of them is from a different k-hop path that may pass different messages to the root node . We assume a node u at any k-hop preserves its initial information prior to receiving information from its child nodes . With this assumption , a node ’ s features are updated by aggregating its child nodes and its own initial features in the tree representation . This tree representation of graphs leads to a new graph learning architecture - Graph Tree Networks ( GTreeNets ) as shown in Appendix A.1 , with the general message passing rule formulated as Equation 1 : hku = fk ( aggregate ( ϕ ( xu ) , { hk+1v , ∀v ∈ Nu } ) ) ( 1 ) where k = L−1 , . . . , 0 . L is the tree depth ( number of propagation layers ) . xu is the vector of input features of node u. zu = ϕ ( xu ) is the transformation of input features of node u , i.e . the initial hidden features for the propagation layer . hku is the vector of hidden features of node u at k-hop , and hLu = zu . fk ( · ) is a hop-wise ( layer-wise ) transformation function such as MLP . Combination of different aggregation function and transformation function in Equation 1 derives various graph learning models under the GTreeNet architecture . We propose two graph tree models in Section 2.2 and 2.3 : Graph Tree Convolution Network ( GTCN ) and Graph Tree Attention Network ( GTAN ) . 2.1.1 RELATION TO PROPAGATION SCHEME IN GCN-LIKE MODELS . The general message passing rule in GCN-like models such as GCN ( Kipf & Welling , 2017 ) , GAT ( Velickovic et al. , 2018 ) , and GraphSage ( Hamilton et al. , 2017 ) is hnu = fn−1 ( aggregate ( hn−1u , { hn−1v , ∀v ∈ Nu } ) ) ( 2 ) Comparing to Equation 1 , it is notable that the message passing scheme in our proposed GTreeNet is essentially different from the one adopted by the GCN-like models . In GTreeNet , we update a node ’ s hidden features by aggregating its neighbor ’ s hidden features and its own initial features . While in GCN-like models , a node ’ s hidden features are updated by aggregating its neighbor ’ s hidden features and its own hidden features from the last propagation layer . Our propagation scheme preserves the local information when stacking multiple propagation layers while the propagation scheme in GCNlike models does not . Note that we use n to denote the propagation layer number in Equation 2 to avoid ambiguities . 2.2 GRAPH TREE CONVOLUTION NETWORK . The propagation rule of our proposed Graph Tree Convolution Network is hku = ∑ v∈Nu Âuvh k+1 v + Âuuzu ( 3 ) where Â = D̃− 1 2 ÃD̃− 1 2 is the symmetric normalized adjacency matrix1 . Ã = ( A+ I ) is the adjacency matrix with added self-loops . D̃ is the degree matrix with self-loops where D̃uu =∑ v Ãuv . zu = MLP ( xu ) . 1We find that using Â = D̃−1Ã yields similar performance . As compared to the general propagation rule in GTreeNet shown in Equation 1 , our GTCN model uses an MLP as the transformation function ϕ ( · ) , an identity function as fk ( · ) , and a weighted sum function as the aggregation function with weights being the elements of the normalized adjacency matrix . The mathematical description of our GTCN model is then formulated as Z = MLP ( X ) HL = Z Hk = A1H k+1 +A2Z , k = L− 1 , L− 2 , . . . , 0 Yout = softmax ( H0W ) ( 4 ) where Z ∈ RN×F is the initial hidden feature map obtained by applying an MLP to the input feature map X . Hk ∈ RN×F is the hidden feature map at the kth hop . A1 and A2 are the non-diagonal and diagonal component of the normalized adjacency matrix , respectively . W ∈ RF×C is the trainable weight matrix . Note that our model separates the transformation from propagation with no trainable parameters required in the propagation layers . By aggregating a node ’ s neighbor information with its own initial information , we always preserve the local information of a node . These two characteristics make our model capable of going deep without comprising performance as demonstrated in Section 3.2 .

Summary: The paper studies the task of network embedding. The authors propose a Graph Tree Networks (GTree), a deep graph neural network architecture that originates from the tree representation of the graphs. In particular, the proposed approach conduct message passing upward from the leaf nodes to the root node to update the root node’s hidden features. Experiment validates the effectiveness of the proposed model,