DropEdge: Towards Deep Graph Convolutional Networks on Node Classification

1 INTRODUCTION . Graph Convolutional Networks ( GCNs ) , which exploit message passing or equivalently certain neighborhood aggregation function to extract high-level features from a node as well as its neighborhoods , have boosted the state-of-the-arts for a variety of tasks on graphs , such as node classification ( Bhagat et al. , 2011 ; Zhang et al. , 2018 ) , social recommendation ( Freeman , 2000 ; Perozzi et al. , 2014 ) , and link prediction ( Liben-Nowell & Kleinberg , 2007 ) to name some . In other words , GCNs have been becoming one of the most crucial tools for graph representation learning . Yet , when we revisit typical GCNs on node classification ( Kipf & Welling , 2017 ) , they are usually shallow ( e.g . the number of the layers is 21 ) . Inspired from the success of deep CNNs on image classification , several attempts have been proposed to explore how to build deep GCNs towards node classification ( Kipf & Welling , 2017 ; Li et al. , 2018a ; Xu et al. , 2018a ; Li et al. , 2019 ) ; nevertheless , none of them delivers sufficiently expressive architecture . The motivation of this paper is to analyze the very factors that impede deeper GCNs to perform promisingly , and develop method to address them . We begin by investigating two factors : over-fitting and over-smoothing . Over-fitting comes from the case when we utilize an over-parametric model to fit a distribution with limited training data , where the model we learn fits the training data very well but generalizes poorly to the testing data . It does exist if we apply a deep GCN on small graphs ( see 4-layer GCN on Cora in Figure 1 ) . Over-smoothing , towards the other extreme , makes training a very deep GCN difficult . As first introduced by Li et al . ( 2018a ) and further explained in Wu et al . ( 2019 ) ; Xu et al . ( 2018a ) ; Klicpera et al . ( 2019 ) , graph convolutions essentially push representations of adjacent nodes mixed with each ∗Wenbing Huang is the corresponding author . 1When counting the number of layers ( or network depth ) of GCN , this paper does not involve the input layer . other , such that , if extremely we go with an infinite number of layers , all nodes ’ representations will converge to a stationary point , making them unrelated to the input features and leading to vanishing gradients . We call this phenomenon as over-smoothing of node features . To illustrate its influence , we have conducted an example experiment with 8-layer GCN in Figure 1 , in which the training of such a deep GCN is observed to converge poorly . Both of the above two issues can be alleviated , using the proposed method , DropEdge . The term “ DropEdge ” refers to randomly dropping out certain rate of edges of the input graph for each training time . There are several benefits in applying DropEdge for the GCN training ( see the experimental improvements by DropEdge in Figure 1 ) . First , DropEdge can be considered as a data augmentation technique . By DropEdge , we are actually generating different random deformed copies of the original graph ; as such , we augment the randomness and the diversity of the input data , thus better capable of preventing over-fitting . Second , DropEdge can also be treated as a message passing reducer . In GCNs , the message passing between adjacent nodes is conducted along edge paths . Removing certain edges is making node connections more sparse , and hence avoiding over-smoothing to some extent when GCN goes very deep . Indeed , as we will draw theoretically in this paper , DropEdge either reduces the convergence speed of over-smoothing or relieves the information loss caused by it . We are also aware that the dense connections employed by JKNet ( Xu et al. , 2018a ) are another kind of tools that can potentially prevent over-smoothing . In its formulation , JKNet densely connects each hidden layer to the top one , hence the feature mappings in lower layers that are hardly affected by over-smoothing are still maintained . Interestingly and promisingly , we find that the performance of JKNet can be promoted further if it is utilized along with our DropEdge . Actually , our DropEdge—as a flexible and general technique—is able to enhance the performance of various popular backbone networks on several benchmarks , including GCN ( Kipf & Welling , 2017 ) , ResGCN ( Li et al. , 2019 ) , JKNet ( Xu et al. , 2018a ) , and GraphSAGE ( Hamilton et al. , 2017 ) . We provide detailed evaluations in the experiments . 2 RELATED WORK . GCNs Inspired by the huge success of CNNs in computer vision , a large number of methods come redefining the notion of convolution on graphs under the umbrella of GCNs . The first prominent research on GCNs is presented in Bruna et al . ( 2013 ) , which develops graph convolution based on spectral graph theory . Later , Kipf & Welling ( 2017 ) ; Defferrard et al . ( 2016 ) ; Henaff et al . ( 2015 ) ; Li et al . ( 2018b ) ; Levie et al . ( 2017 ) apply improvements , extensions , and approximations on spectralbased GCNs . To address the scalability issue of spectral-based GCNs on large graphs , spatial-based GCNs have been rapidly developed ( Hamilton et al. , 2017 ; Monti et al. , 2017 ; Niepert et al. , 2016 ; 2To check the efficacy of DropEdge more clearly , here we have removed bias in all GCN layers , while for the experiments in § 5 , the bias are kept . Gao et al. , 2018 ) . These methods directly perform convolution in the graph domain by aggregating the information from neighbor nodes . Recently , several sampling-based methods have been proposed for fast graph representation learning , including the node-wise sampling methods ( Hamilton et al. , 2017 ) , the layer-wise approach ( Chen et al. , 2018 ) and its layer-dependent variant ( Huang et al. , 2018 ) . Specifically , GAT ( Velickovic et al. , 2018 ) has discussed applying dropout on edge attentions . While it actually is a post-conducted version of DropEdge before attention computation , the relation to over-smoothing is never explored in Velickovic et al . ( 2018 ) . In our paper , however , we have formally presented the formulation of DropEdge and provided rigorous theoretical justification of its benefit in alleviating over-smoothing . We also carried out extensive experiments by imposing DropEdge on several popular backbones . One additional point is that we further conduct adjacency normalization after dropping edges , which , even simple , is able to make it much easier to converge during training and reduce gradient vanish as the number of layers grows . Deep GCNs Despite the fruitful progress , most previous works only focus on shallow GCNs while the deeper extension is seldom discussed . The attempt for building deep GCNs is dated back to the GCN paper ( Kipf & Welling , 2017 ) , where the residual mechanism is applied ; unexpectedly , as shown in their experiments , residual GCNs still perform worse when the depth is 3 and beyond . The authors in Li et al . ( 2018a ) first point out the main difficulty in constructing deep networks lying in over-smoothing , but unfortunately , they never propose any method to address it . The follow-up study ( Klicpera et al. , 2019 ) solves over-smoothing by using personalized PageRank that additionally involves the rooted node into the message passing loop ; however , the accuracy is still observed to decrease when the depth increases from 2 . JKNet ( Xu et al. , 2018a ) employs dense connections for multi-hop message passing which is compatible with DropEdge for formulating deep GCNs . Oono & Suzuki ( 2019 ) theoretically prove that the node features of deep GCNs will converge to a subspace and incur information loss . It generalizes the conclusion in Li et al . ( 2018a ) by further considering the ReLu function and convolution filters . Our interpretations on why DropEdge can impede over-smoothing is based on the concepts proposed by Oono & Suzuki ( 2019 ) . A recent method ( Li et al. , 2019 ) has incorporated residual layers , dense connections and dilated convolutions into GCNs to facilitate the development of deep architectures . Nevertheless , this model is targeted on graph-level classification ( i.e . point cloud segmentation ) , where the data points are graphs and naturally disconnected from each other . In our task for node classification , the samples are nodes and they all couple with each other , thus the over-smoothing issue is more demanded to be addressed . By leveraging DropEdge , we are able to relieve over-smoothing , and derive more enhanced deep GCNs on node classification . 3 NOTATIONS AND PRELIMINARIES . Notations . Let G = ( V , E ) represent the input graph of size N with nodes vi ∈ V and edges ( vi , vj ) ∈ E . The node features are denoted as X = { x1 , · · · , xN } ∈ RN×C , and the adjacency matrix is defined as A ∈ RN×N which associates each edge ( vi , vj ) with its element Aij . The node degrees are given by d = { d1 , · · · , dN } where di computes the sum of edge weights connected to node i . We define D as the degree matrix whose diagonal elements are obtained from d. GCN is originally developed by Kipf & Welling ( 2017 ) . The feed forward propagation in GCN is recursively conducted as H ( l+1 ) = σ ( ÂH ( l ) W ( l ) ) , ( 1 ) where H ( l+1 ) = { h ( l+1 ) 1 , · · · , h ( l+1 ) N } are the hidden vectors of the l-th layer with h ( l ) i as the hidden feature for node i ; Â = D̂−1/2 ( A+ I ) D̂−1/2 is the re-normalization of the adjacency matrix , and D̂ is the corresponding degree matrix of A+ I ; σ ( · ) is a nonlinear function , i.e . the ReLu function ; and W ( l ) ∈ RCl×Cl−1 is the filter matrix in the l-th layer with Cl refers to the size of l-th hidden layer . We denote one-layer GCN computed by Equation 1 as Graph Convolutional Layer ( GCL ) in what follows . 4 OUR METHOD : DROPEDGE . This section first introduces the methodology of the DropEdge technique as well as its layer-wise variant where the adjacency matrix for each GCN layer is perturbed individually . We also explain how the proposed DropEdge can prevent over-fitting and over-smoothing in generic GCNs . Particularly for over-smoothing , we provide its mathematical definition and theoretical derivations on showing the benefits of DropEdge . 4.1 METHODOLOGY . At each training epoch , the DropEdge technique drops out a certain rate of edges of the input graph by random . Formally , it randomly enforces V p non-zero elements of the adjacency matrix A to be zeros , where V is the total number of edges and p is the dropping rate . If we denote the resulting adjacency matrix as Adrop , then its relation with A becomes Adrop = A−A′ , ( 2 ) where A′ is a sparse matrix expanded by a random subset of size V p from original edges E . Following the idea of Kipf & Welling ( 2017 ) , we also perform the re-normalization trick on Adrop , leading to Âdrop . We replace Â with Âdrop in Equation 1 for propagation and training . When validation and testing , DropEdge is not utilized . Preventing over-fitting . DropEdge produces varying perturbations of the graph connections . As a result , it generates different random deformations of the input data and can be regarded as a data augmentation skill for graphs . To explain why this is valid , we provide an intuitive understanding here . The key in GCNs is to aggregate neighbors ’ information for each node , which can be understood as a weighted sum of the neighbor features ( the weights are associated with the edges ) . From the perspective of neighbor aggregation , DropEdge enables a random subset aggregation instead of the full aggregation during GNN training . Statistically , DropEdge only changes the expectation of the neighbor aggregation up to a multiplier p , if we drop edges with probability p. This multiplier will be actually removed after weights normalization , which is often the case in practice . Therefore , DropEdge does not change the expectation of neighbor aggregation and is an unbiased data augmentation technique for GNN training , similar to typical image augmentation skills ( e.g . rotation , cropping and flapping ) that are capable of hindering over-fitting in training CNNs . We will provide experimental validations in § 5.1 . Layer-Wise DropEdge . The above formulation of DropEdge is one-shot with all layers sharing the same perturbed adjacency matrix . Indeed , we can perform DropEdge for each individual layer . Specifically , we obtain Â ( l ) drop by independently computing Equation 2 for each l-th layer . Different layer could have different adjacency matrix Â ( l ) drop . Such layer-wise version brings in more randomness and deformations of the original data , and we will experimentally compare its performance with the original DropEdge in § 5.2 . Over-smoothing is another obstacle of training deep GCNs , and we will detail how DropEdge can address it to some extent in the next section . For simplicity , the following derivations assume all GCLs share the same perturbed adjacency matrix , and we will leave the discussion on layer-wise DropEdge for future exploration .

Summary: The authors propose a simple but effective strategy that aims to alleviate not only overfitting, but also feature degradation (oversmoothing) in deep graph convolutional networks (GCNs).  Inspired by dropout in traditional MLPs and convnets, the authors clearly motivate their contribution in terms of alleviating both overfitting and oversmoothing, which are problems established both in previous literature as well as validated empirically by the authors. Ultimately, the authors provide solid empirical evidence that, while a bit heuristic, their method is effective at alleviating at least partially the issues of overfitting and oversmoothing.