On Provable Benefits of Depth in Training Graph Convolutional Networks

1 Introduction . In recent years , Graph Convolutional Networks ( GCNs ) have achieved state-of-the-art performance in dealing with graph-structured applications , including social networks [ 28 , 21 , 51 , 12 , 44 ] , traffic prediction [ 10 , 45 , 34 , 31 ] , knowledge graphs [ 52 , 53 , 43 ] , drug reaction [ 14 , 16 ] and recommendation system [ 2 , 60 ] . Despite the success of GCNs , applying a shallow GCN model that only uses the information of a very limited neighborhood on a large sparse graph has shown to be not effective [ 23 , 6 , 20 , 9 , 46 ] . As a result , a deeper GCN model would be desirable to reach and aggregate information from farther neighbors . The inefficiency of shallow GCNs is exacerbated even further when the labeled nodes compared to graph size is negligible , as a shallow GCN can not sufficiently propagate the label information to the entire graph with only a few available labels [ 35 ] . Although a deeper GCN is preferred to perceive more graph structure information , unlike traditional deep neural networks , it has been pointed out that deeper GCNs potentially suffer from oversmoothing [ 35 , 41 , 26 , 4 , 58 ] , vanishing/exploding gradients [ 33 ] , over-squashing [ 1 ] , and training difficulties [ 62 , 38 ] , which significantly affect the performance of GCNs as the depth increases . Among these , the most widely accepted reason is “ over-smoothing ” , which is referred to as a phenomenon due to applying multiple graph convolutions such that all node embeddings converge to a single subspace ( or vector ) and leads to indistinguishable node representations . The conventional wisdom is that adding to the number of layers causes over-smoothing , which impairs the expressiveness power of GCNs and consequently leads to a poor training accuracy . However , we observe that there exists a discrepancy between theoretical understanding of their inherent capabilities 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . and practical performances . According to the definition of over-smoothing that the node representation becomes indistinguishable as GCNs goes deeper , the classifier has difficulty assigning the correct label for each node if over-smoothing happens . As a result , the training accuracy is expected to be decreasing as the number of layers increases . However , as shown in Figure 1 , GCNs are capable of achieving high training accuracy regardless of the number of layers . But , as it can be observed , deeper GCNs require more training iterations to reach a high training accuracy , and its generalization performance on evaluation set decreases as the number of layers increases . This observation suggests that the performance degradation is likely due to inappropriate training rather than the low expressive power caused by over-smoothing . Otherwise , a low expressiveness model can not achieve almost perfect training accuracy simply by proper training tricks alone.1 Indeed , recent years significant advances have been witnessed on tweaking the model architecture to overcome the training difficulties in deeper GCN models and achieve good generalization performance [ 38 , 6 , 33 , 62 ] . Contributions . Motivated by aforementioned observation , i.e. , still achieving high training accuracy when trained properly but poor generalization performance , we aim at answering two fundamental questions in this paper : Q1 : Does increasing depth really impair the expressiveness power of GCNs ? In Section 4 , we argue that there exists a discrepancy between over-smoothing based theoretical results and the practical capabilities of deep GCN models , demonstrating that over-smoothing is not the key factor that leads to the performance degradation in deeper GCNs . In particular , we mathematically show that over-smoothing [ 41 , 26 , 35 , 4 ] is mainly an artifact of theoretical analysis and simplifications made in analysis . Indeed , by characterizing the representational capacity of GCNs via Weisfeiler-Lehman ( WL ) graph isomorphism test [ 39 , 55 ] , we show that deeper GCN model is at least as expressive as the shallow GCN model , the deeper GCN models can distinguish nodes with a different neighborhood that the shallow GCN can not distinguish , as long as the GCNs are properly trained . Besides , we theoretically show that more training iterations is sufficient ( but not necessary due to the assumptions made in our theoretical analysis ) for a deeper model to achieve the same training error as the shallow ones , which further suggests the poor training error in deep GCN training is most likely due to inappropriate training . Q2 : If expressive , why then deep GCNs generalize poorly ? In Section 5 , in order to understand the performance degradation phenomenon in deep GCNs during the evaluation phase , we give a novel generalization analysis on GCNs and its variants ( e.g. , ResGCN , APPNP , and GCNII ) under the semi-supervised setting for the node classification task . We show that the generalization gap of GCNs is governed by the number of training iterations , largest node degree , the largest singular value of weight matrices , and the number of layers . In particular , our result suggests that a deeper GCN model requires more iterations of training and optimization tricks to converge ( e.g. , adding skip-connections ) , which leads to a poor generalization . More interestingly , our generalization analysis shows that most of the so-called methods to solve oversmoothing [ 47 , 61 , 6 , 29 ] can greatly improve the generalization ability of the model , therefore results in a deeper model . 1The results in Figure 1 can be reproduced by removing both the dropout and weight decay operation . These two augmentations are designed to improve the generalization ability ( i.e. , validation/testing accuracy ) of the model but might hurt the training accuracy , because of the randomness and also an extra regularization term on the model parameters which are introduced during training . A simple experiment using DGL can be found here . The aforementioned findings naturally lead to the algorithmic contribution of this paper . In Section 6 , we present a novel framework , Decoupled GCN ( DGCN ) , that is capable of training deeper GCNs and can significantly improve the generalization performance . The main idea is to isolate the expressive power from generalization ability by decoupling the weight parameters from feature propagation . In Section 7 , we conduct experiments on the synthetic and real-world datasets to validate the correctness of the theoretical analysis and the advantages of DGCN over baseline methods . 2 Related works . Expressivity of GCNs . Existing results on expressive power of GCNs are mixed , [ 39 , 37 , 7 , 5 ] argue that deeper model has higher expressive power but [ 41 , 26 , 25 ] have completely opposite result . On the one hand , [ 39 ] shows a deeper GCN is as least as powerful as the shallow one in terms of distinguishing non-isomorphic graphs . [ 37 ] shows that deep and wide GCNs is Turing universal , however , GCNs lose power when their depth and width are restricted . [ 7 ] and [ 5 ] measure the expressive power of GCNs via its subgraph counting capability and attribute walks , and both show the expressive power of GCNs grows exponentially with the GCN depth . On the other hand , [ 25 ] studies the infinity wide GCNs and shows that the covariance matrix of its outputs converges to a constant matrix at an exponential rate . [ 41 , 26 ] characterize the expressive power using the distance between node embeddings to a node feature agnostic subspace and show the distance is decreasing as the number of layers increases . Details are deferred to Section 4 and Appendix B . Such contradictory results motivate us to rethink the role of over-smoothing on expressiveness . Generalization analysis of GCNs . In recent years , many papers are working on the generalization of GCNs using uniform stability [ 50 , 63 ] , Neural Tangent Kernel [ 15 ] , VC-dimension [ 49 ] , Rademacher complexity [ 18 , 42 ] , algorithm alignment [ 56 , 57 ] and PAC-Bayesian [ 36 ] . Existing works only focus on a specific GCN structure , which can not be used to understand the impact of GCN structures on its generalization ability . The most closely related to ours is [ 50 ] , where they analyze the stability of the single-layer GCN model , and show that the stability of GCN depends on the largest absolute eigenvalue of its Laplacian matrix . However , their result is under the inductive learning setting and extending the results to the multi-layer GCNs with different structures is non-trivial . Literature with similar observations . Most recently , several works have similar observations on the over-smoothing issue to ours . [ 62 ] argues that the main factors to performance degradation are vanishing gradient , training instability , and over-fitting , rather than over-smoothing , and proposes a node embedding normalization heuristic to alleviate the aforementioned issues . [ 38 ] argues that the performance degradation is mainly due to the training difficulty , and proposes a different graph Laplacian formulation , weight parameter initialization , and skip-connections to improve the training difficulty . [ 59 ] argues that deep GCNs can learn to overcome the over-smoothing issue during training , and the key factor of performance degradation is over-fitting , and proposes a node embedding normalization method to help deep GCNs overcome the over-smoothing issue . [ 30 ] improves the generalization ability of GCNs by using adversarial training and results in a consistent improvement than all GCNs baseline models without adversarial training . All aforementioned literature only gives heuristic explanations based on the empirical results , and do not provide theoretical arguments . 3 Preliminaries . Notations and setup . We consider the semi-supervised node classification problem , where a selfconnected graph G = ( V , E ) withN = |V| nodes is given in which each node i ∈ V is associated with a feature vector xi ∈ Rd0 , and only a subset of nodes Vtrain ⊂ V are labeled , i.e. , yi ∈ { 1 , . . . , |C| } for each i ∈ Vtrain and C is the set of all candidate classes . Let A ∈ RN×N denote the adjacency matrix and D ∈ RN×N denote the corresponding degree matrix with Di , i = deg ( i ) and Di , j = 0 if i 6= j . Then , the propagation matrix ( using the Laplacian matrix defined in [ 28 ] ) is computed as P = D−1/2AD−1/2 . Our goal is to learn a GCN model using the node features for all nodes { xi } i∈V and node labels for the training set nodes { yi } i∈Vtrain , and expect it generalizes well on the unlabeled node set Vtest = V \ Vtrain . GCN architectures . In this paper , we consider the following architectures for training GCNs : There exist a fully connected layer after node feature No fully connected layer after node feature Cora • Vanilla GCN [ 28 ] computes node embeddings by H ( ` ) = σ ( PH ( ` −1 ) W ( ` ) ) , where H ( ` ) = { h ( ` ) i } Ni=1 is the ` th layer node embedding matrix , h ( ` ) i ∈ Rd ` is the embedding of ith node , W ( ` ) ∈ Rd ` ×d ` −1 is the ` th layer weight matrix , and σ ( · ) is the ReLU activation . • ResGCN [ 33 ] solves the vanishing gradient issue by adding skip-connections between adjacency layers . More specifically , ResGCN computes node embeddings by H ( ` ) = σ ( PH ( ` −1 ) W ( ` ) ) + H ( ` −1 ) , where node embeddings of the previous layer is added to the output of the current layer to facilitate the training of deeper GCN models . • APPNP [ 29 ] adds skip-connections from the input layer to each hidden layer to preserve the feature information . APPNP computes node embeddings by H ( ` ) = α ` PH ( ` −1 ) + ( 1−α ` ) H ( 0 ) W , where α ` ∈ [ 0 , 1 ] balances the amount of information preserved at each layer . By decoupling feature transformation and propagation , APPNP can aggregate information from multi-hop neighbors without significantly increasing the computation complexity . • GCNII [ 6 ] improves the capacity of APPNP by adding non-linearty and weight matrix at each individual layer . GCNII computes node embeddings by H ( ` ) = σ ( ( α ` PH ( ` −1 ) + ( 1 − α ` ) H ( 0 ) ) W̄ ( ` ) ) , where W̄ ( ` ) = β ` W ( ` ) + ( 1− β ` ) I , constant α ` same as APPNP , and constant β ` ∈ [ 0 , 1 ] restricts the power of ` th layer parameters .

Summary: The deals with semi-supervised learning with graphs using graph neural networks. Specifically, it investigates the benefits of depth both from a (learning) theory as well as an empirical standpoint.   The key contributions are: 1. Questioning the common folklore belief that deep GNNs are not expressive due to "over-smoothing" 2. Studying the generalization performance of GNNs using uniform stability theory  3. Introducing  "Decoupled GCNs" which learn linear combinations of single-layer GNNs (using powers of the Laplacian). 4. Experimental study validating the theoretical results.