Evaluating Deep Graph Neural Networks

1 INTRODUCTION . The recent success of Graph Neural Networks ( GNNs ) ( Zhang et al. , 2020 ) has boosted researches on knowledge discovery and data mining on graph data . Designed for graph-structured data , GNNs provide a universal way to tackle node-level , edge-level , and graph-level tasks , including social network analysis ( Qiu et al. , 2018 ; Fan et al. , 2019 ; Huang et al. , 2021 ) , chemistry and biology ( Dai et al. , 2019 ; Bradshaw et al. , 2019 ; Do et al. , 2019 ) , recommendation ( Monti et al. , 2017 ; Wu et al. , 2020 ; Yin et al. , 2019 ) , natural language processing ( Bastings et al. , 2017 ; Wu et al. , 2021 ; Vashishth et al. , 2020 ) , and computer vision ( Qi et al. , 2018 ; Shi et al. , 2019 ; Sarlin et al. , 2020 ) . The key to the success of most GNNs lies in the graph convolution operation , which propagates neighbor information to the center node in an iterative manner ( Wu et al. , 2019 ) . The graph convolution operation can be further decomposed into two sequential operations : embedding propagation ( EP ) and embedding transformation ( ET ) . The EP operation can be viewed as a special form of Laplacian smoothing ( NT & Maehara , 2019 ) , which combines the embeddings of a node itself and its one-hop neighbors . The embeddings of nodes within the same connected component would become similar after applying the smoothing operation , which greatly eases the downstream tasks . The ET operation applies neural networks and transforms the node embeddings to target dimensions . Taking the widely-used Graph Convolutional Network ( GCN ) ( Kipf & Welling , 2016 ) as an example , through stacking k convolution operations ( i.e. , layers ) , each node in GCN can utilize the information from nodes within its k-hop neighborhood , and thus improve the predictive accuracy by getting more unlabeled nodes involved in the training process . Despite the remarkable success , simply stacking many graph convolution operations leads to massive performance degradation . As a result , most GNNs today only have shallow architectures ( e.g. , 2 or 3 layers ) , which limits their exploitation of deep structural information . Concretely , under the semi-supervised setting where only a few labels are given , shallow GNNs can utilize only a small percentage of nodes during model training , leading to sub-optimal node classification accuracy . To alleviate the problem that GNNs can not go deep , many researches have been proposed , and they attribute the performance degradation of deep GNNs to several reasons . Among the suggested reasons , most existing works ( Feng et al. , 2020 ; Chen et al. , 2020a ; Zhao & Akoglu , 2020 ; Godwin et al. , 2021 ; Rong et al. , 2019 ; Zeng et al. , 2020a ; Min et al. , 2020 ; Chamberlain et al. , 2021 ; Chien et al. , 2021 ; Zhou et al. , 2020a ; Hou et al. , 2019 ; Beaini et al. , 2021 ; Yan et al. , 2021 ; Cai & Wang , 2020 ) consider the over-smoothing issue as the major cause of performance degradation of deep GNNs . Notice that the EP operation smooths the node embeddings , i.e. , making nodes within the same connected component similar . If a GNN is stacked with a large number of graph convolution operations , the output embeddings might be over-smoothed , i.e. , nodes within the same connected component become indistinguishable . Questions Investigated . In this paper , we dive deep into the problem of why most existing GNNs can not go deep and try to present answers to the following two key questions : Q1 : What actually limits the deep stacking of convolution operations in GNN designs ? Q2 : How can we design deep GNNs with the help of the findings from the experimental analysis and outperform the state-of-the-art GNNs ? Contributions . To answer the above research questions , we first conduct a comprehensive evaluation to revise the over-smoothing issue and identify the root cause of performance degradation of most existing GNNs when they go deep . Based on the above analysis , we obtain helpful insights and guidelines to design deep GNNs . Our main contributions are summarized as follows . C1 : We clarify the concept of model depth by separating and considering the two different depths when designing deep GNNs : the propagation depth Dp and the transformation depth Dt . Through experimental evaluations , we find that large Dp leads to the over-smoothing issue whereas large Dt leads to the model degradation issue in the current GNN models . Moreover , we observed that the latter usually happens much earlier than the former as Dp and Dt increase at the same speed . Thus , the model degradation issue introduced by large Dt is the true root cause for the failure of deep GNNs . C2 : To design models that support largeDp , we propose a node-adaptive combination mechanism for combining propagated features under EP operations of different steps . To support large Dt , we add residual connections between ET operations to alleviate the model degradation issue . Further , we present Deep Graph Multi-Layer Perceptron ( DGMLP ) , a novel approach that adopts the composition of the above mentioned two mechanisms to successfully support both large Dp and large Dt based on our findings from the experimental analysis . We validate the effectiveness of DGMLP on six public datasets and the Industry dataset from the real industrial environment . Experimental results demonstrate that DGMLP outperforms the SOTA GNNs while maintaining high scalability and efficiency . To the best of our knowledge , this paper is the first to conduct an experimental evaluation that identifies the major reason why most existing GNNs can not go deep . Our findings and the derived guidelines open up a new perspective on designing deep GNNs for graph-structured data . 2 PRELIMINARY . In this section , we first explain the problem formulation . Then we introduce Embedding Propagation ( EP ) and Embedding Transformation ( ET ) in the graph convolution operation in detail . 2.1 PROBLEM FORMALIZATION . In this paper , we consider an undirected graph G = ( V , E ) with |V| = N nodes and |E| =M edges . A is the adjacency matrix of G , weighted or not . Each node possibly has a feature vector of size d , stacked up to an N × d matrix X . D = diag ( d1 , d2 , · · · , dn ) ∈ RN×N denotes the degree matrix of A , where di = ∑ j∈V Aij is the degree of node i . In this paper , we focus on the semi-supervised node classification task . Suppose Vl is the labeled node set , the goal is to predict the labels for nodes in the unlabeled set Vu under the limited supervision of labels for nodes in Vl . 2.2 CONVOLUTION ON GRAPHS . Graph Convolution . Based on the intuitive assumption that locally connected nodes are likely to have the same label ( McPherson et al. , 2001 ) , GNN iteratively propagates the information of each node to its adjacent nodes . For example , each graph convolution operation in GCN firstly propagates the node embeddings to their neighborhoods and then transforms their propagated node embeddings : X ( k+1 ) = σ ( ÂX ( k ) W ( k ) ) , Â = D̃ 1 2 ÃD̃− 1 2 , ( 1 ) where X ( k ) and X ( k+1 ) are the node embedding matrices at layer k and k + 1 , respectively . Ã = A + IN is the adjacency matrix of the undirected graph G with self loops added , where IN is the identity matrix . Â is the normalized adjacency matrix , and D̃ is its corresponding degree matrix . By setting different r in Â = D̃r−1ÃD̃−r , different normalization strategies can be employed , such as the symmetric normalized matrix D̃− 1 2 ÃD̃− 1 2 ( Kipf & Welling , 2016 ) , the random walk transition probability matrix D̃−1Ã ( Xu et al. , 2018 ) , and the reverse random walk transition probability matrix ÃD̃−1 ( Zeng et al. , 2020b ) . We adopt Â = D̃− 1 2 ÃD̃− 1 2 in this work . EP and ET Operations . Each graph convolution operation in GNNs can be decomposed into two sequential operations : Embedding Propagation ( EP ) and Embedding Transformation ( ET ) . This decomposition naturally leads to two corresponding GNN depths : propagation depth Dp and transformation depth Dt . Concretely , GNN first executes EP , which generates smoothed features by multiplying the normalized adjacency matrix Â with the node embedding matrix X : EP ( X ) = ÂX . ( 2 ) Then , the smoothed features X̂ = EP ( X ) will be transformed with the learnable transformation matrix W and the activation function σ ( · ) : ET ( X̂ ) = σ ( X̂W ) . ( 3 ) Fig . 1 shows the framework of a two-layer GCN . To note that , GCN will degrade to MLP if Â is the identity matrix , which is equal to removing the EP operation in all GCN layers . More detailed analysis and classification of current GNN approaches can be found in Appendix A . 3 SMOOTHNESS MEASUREMENT . In Eq . 2 , each time Â multiplies with X , information one more hop away can be acquired for each node . Thus , in order to fully leverage high-order neighborhood information , a series of multiplications of ÂX , i.e. , the EP operation , have to be carried out , which means stacking multiple GNN layers . However , if we execute ÂX numerous times , the node embeddings within the same connected component would reach a stationary state , leading to indistinguishable node embeddings ( i.e. , oversmoothing issue ) . Concretely , when adopting Â = D̃r−1ÃD̃−r , Â∞ follows Â∞i , j = ( di + 1 ) r ( dj + 1 ) 1−r 2m+ n , ( 4 ) which shows that after infinite times of multiplication , the influence from node i to j is only determined by the degrees of them . Under this scenario , the neighborhood information is fully corrupted , resulting in catastrophic node classification accuracy . As the over-smoothing issue is only introduced by the EP operation rather than the ET operation , here we introduce a new metric , “ Node Smoothness Level ( NSL ) ” , to evaluate the smoothness of a node after k steps of EP operation . Suppose X ( 0 ) = X is the original node feature matrix , and X ( k ) = ÂkX ( 0 ) is the smoothed features after k times of EP operation . Definition 3.1 ( Node Smoothing Level ) . The Node Smoothing Level NSLv ( k ) parameterized by node v and the EP steps , k , is defined as : α = Sim ( xkv , x 0 v ) , β = Sim ( x k v , x ∞ v ) , NSLv ( k ) = α ∗ ( 1− β ) , ( 5 ) where xkv is the smoothed feature of node v after k steps of EP operation , x 0 v represents node v ’ s original feature , and x∞v represents node v ’ s feature at stationary state . Sim ( · ) is a similarity function , being the cosine similarity in the following discussion . Further , the “ Graph Smoothing Level ” ( GSL ) parameterized by the EP steps , k , is defined as : GSL ( k ) = 1 N ∑ v∈V NSLv ( k ) . ( 6 ) Smaller GSL ( k ) means that X ( k ) is more likely to forget the original node feature information X ( 0 ) after k steps of EP operation and has a higher risk of the over-smoothing issue .

Summary: This work performs a systematic study to analyze the main issues of the difficulty in training deep GNNs by disentangling the effects of embedding propagation (EP) and embedding transformation (ET). They find that the large $D_t$ is the root cause for the failure of deep GNNs. Node-adaptive combination mechanism and residual connections between ET operations are proposed to train deep GNNs.