On Provable Benefits of Depth in Training Graph Convolutional Networks

1 Introduction . In recent years , Graph Convolutional Networks ( GCNs ) have achieved state-of-the-art performance in dealing with graph-structured applications , including social networks [ 28 , 21 , 51 , 12 , 44 ] , traffic prediction [ 10 , 45 , 34 , 31 ] , knowledge graphs [ 52 , 53 , 43 ] , drug reaction [ 14 , 16 ] and recommendation system [ 2 , 60 ] . Despite the success of GCNs , applying a shallow GCN model that only uses the information of a very limited neighborhood on a large sparse graph has shown to be not effective [ 23 , 6 , 20 , 9 , 46 ] . As a result , a deeper GCN model would be desirable to reach and aggregate information from farther neighbors . The inefficiency of shallow GCNs is exacerbated even further when the labeled nodes compared to graph size is negligible , as a shallow GCN can not sufficiently propagate the label information to the entire graph with only a few available labels [ 35 ] . Although a deeper GCN is preferred to perceive more graph structure information , unlike traditional deep neural networks , it has been pointed out that deeper GCNs potentially suffer from oversmoothing [ 35 , 41 , 26 , 4 , 58 ] , vanishing/exploding gradients [ 33 ] , over-squashing [ 1 ] , and training difficulties [ 62 , 38 ] , which significantly affect the performance of GCNs as the depth increases . Among these , the most widely accepted reason is “ over-smoothing ” , which is referred to as a phenomenon due to applying multiple graph convolutions such that all node embeddings converge to a single subspace ( or vector ) and leads to indistinguishable node representations . The conventional wisdom is that adding to the number of layers causes over-smoothing , which impairs the expressiveness power of GCNs and consequently leads to a poor training accuracy . However , we observe that there exists a discrepancy between theoretical understanding of their inherent capabilities 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . and practical performances . According to the definition of over-smoothing that the node representation becomes indistinguishable as GCNs goes deeper , the classifier has difficulty assigning the correct label for each node if over-smoothing happens . As a result , the training accuracy is expected to be decreasing as the number of layers increases . However , as shown in Figure 1 , GCNs are capable of achieving high training accuracy regardless of the number of layers . But , as it can be observed , deeper GCNs require more training iterations to reach a high training accuracy , and its generalization performance on evaluation set decreases as the number of layers increases . This observation suggests that the performance degradation is likely due to inappropriate training rather than the low expressive power caused by over-smoothing . Otherwise , a low expressiveness model can not achieve almost perfect training accuracy simply by proper training tricks alone.1 Indeed , recent years significant advances have been witnessed on tweaking the model architecture to overcome the training difficulties in deeper GCN models and achieve good generalization performance [ 38 , 6 , 33 , 62 ] . Contributions . Motivated by aforementioned observation , i.e. , still achieving high training accuracy when trained properly but poor generalization performance , we aim at answering two fundamental questions in this paper : Q1 : Does increasing depth really impair the expressiveness power of GCNs ? In Section 4 , we argue that there exists a discrepancy between over-smoothing based theoretical results and the practical capabilities of deep GCN models , demonstrating that over-smoothing is not the key factor that leads to the performance degradation in deeper GCNs . In particular , we mathematically show that over-smoothing [ 41 , 26 , 35 , 4 ] is mainly an artifact of theoretical analysis and simplifications made in analysis . Indeed , by characterizing the representational capacity of GCNs via Weisfeiler-Lehman ( WL ) graph isomorphism test [ 39 , 55 ] , we show that deeper GCN model is at least as expressive as the shallow GCN model , the deeper GCN models can distinguish nodes with a different neighborhood that the shallow GCN can not distinguish , as long as the GCNs are properly trained . Besides , we theoretically show that more training iterations is sufficient ( but not necessary due to the assumptions made in our theoretical analysis ) for a deeper model to achieve the same training error as the shallow ones , which further suggests the poor training error in deep GCN training is most likely due to inappropriate training . Q2 : If expressive , why then deep GCNs generalize poorly ? In Section 5 , in order to understand the performance degradation phenomenon in deep GCNs during the evaluation phase , we give a novel generalization analysis on GCNs and its variants ( e.g. , ResGCN , APPNP , and GCNII ) under the semi-supervised setting for the node classification task . We show that the generalization gap of GCNs is governed by the number of training iterations , largest node degree , the largest singular value of weight matrices , and the number of layers . In particular , our result suggests that a deeper GCN model requires more iterations of training and optimization tricks to converge ( e.g. , adding skip-connections ) , which leads to a poor generalization . More interestingly , our generalization analysis shows that most of the so-called methods to solve oversmoothing [ 47 , 61 , 6 , 29 ] can greatly improve the generalization ability of the model , therefore results in a deeper model . 1The results in Figure 1 can be reproduced by removing both the dropout and weight decay operation . These two augmentations are designed to improve the generalization ability ( i.e. , validation/testing accuracy ) of the model but might hurt the training accuracy , because of the randomness and also an extra regularization term on the model parameters which are introduced during training . A simple experiment using DGL can be found here . The aforementioned findings naturally lead to the algorithmic contribution of this paper . In Section 6 , we present a novel framework , Decoupled GCN ( DGCN ) , that is capable of training deeper GCNs and can significantly improve the generalization performance . The main idea is to isolate the expressive power from generalization ability by decoupling the weight parameters from feature propagation . In Section 7 , we conduct experiments on the synthetic and real-world datasets to validate the correctness of the theoretical analysis and the advantages of DGCN over baseline methods . 2 Related works . Expressivity of GCNs . Existing results on expressive power of GCNs are mixed , [ 39 , 37 , 7 , 5 ] argue that deeper model has higher expressive power but [ 41 , 26 , 25 ] have completely opposite result . On the one hand , [ 39 ] shows a deeper GCN is as least as powerful as the shallow one in terms of distinguishing non-isomorphic graphs . [ 37 ] shows that deep and wide GCNs is Turing universal , however , GCNs lose power when their depth and width are restricted . [ 7 ] and [ 5 ] measure the expressive power of GCNs via its subgraph counting capability and attribute walks , and both show the expressive power of GCNs grows exponentially with the GCN depth . On the other hand , [ 25 ] studies the infinity wide GCNs and shows that the covariance matrix of its outputs converges to a constant matrix at an exponential rate . [ 41 , 26 ] characterize the expressive power using the distance between node embeddings to a node feature agnostic subspace and show the distance is decreasing as the number of layers increases . Details are deferred to Section 4 and Appendix B . Such contradictory results motivate us to rethink the role of over-smoothing on expressiveness . Generalization analysis of GCNs . In recent years , many papers are working on the generalization of GCNs using uniform stability [ 50 , 63 ] , Neural Tangent Kernel [ 15 ] , VC-dimension [ 49 ] , Rademacher complexity [ 18 , 42 ] , algorithm alignment [ 56 , 57 ] and PAC-Bayesian [ 36 ] . Existing works only focus on a specific GCN structure , which can not be used to understand the impact of GCN structures on its generalization ability . The most closely related to ours is [ 50 ] , where they analyze the stability of the single-layer GCN model , and show that the stability of GCN depends on the largest absolute eigenvalue of its Laplacian matrix . However , their result is under the inductive learning setting and extending the results to the multi-layer GCNs with different structures is non-trivial . Literature with similar observations . Most recently , several works have similar observations on the over-smoothing issue to ours . [ 62 ] argues that the main factors to performance degradation are vanishing gradient , training instability , and over-fitting , rather than over-smoothing , and proposes a node embedding normalization heuristic to alleviate the aforementioned issues . [ 38 ] argues that the performance degradation is mainly due to the training difficulty , and proposes a different graph Laplacian formulation , weight parameter initialization , and skip-connections to improve the training difficulty . [ 59 ] argues that deep GCNs can learn to overcome the over-smoothing issue during training , and the key factor of performance degradation is over-fitting , and proposes a node embedding normalization method to help deep GCNs overcome the over-smoothing issue . [ 30 ] improves the generalization ability of GCNs by using adversarial training and results in a consistent improvement than all GCNs baseline models without adversarial training . All aforementioned literature only gives heuristic explanations based on the empirical results , and do not provide theoretical arguments . 3 Preliminaries . Notations and setup . We consider the semi-supervised node classification problem , where a selfconnected graph G = ( V , E ) withN = |V| nodes is given in which each node i ∈ V is associated with a feature vector xi ∈ Rd0 , and only a subset of nodes Vtrain ⊂ V are labeled , i.e. , yi ∈ { 1 , . . . , |C| } for each i ∈ Vtrain and C is the set of all candidate classes . Let A ∈ RN×N denote the adjacency matrix and D ∈ RN×N denote the corresponding degree matrix with Di , i = deg ( i ) and Di , j = 0 if i 6= j . Then , the propagation matrix ( using the Laplacian matrix defined in [ 28 ] ) is computed as P = D−1/2AD−1/2 . Our goal is to learn a GCN model using the node features for all nodes { xi } i∈V and node labels for the training set nodes { yi } i∈Vtrain , and expect it generalizes well on the unlabeled node set Vtest = V \ Vtrain . GCN architectures . In this paper , we consider the following architectures for training GCNs : There exist a fully connected layer after node feature No fully connected layer after node feature Cora • Vanilla GCN [ 28 ] computes node embeddings by H ( ` ) = σ ( PH ( ` −1 ) W ( ` ) ) , where H ( ` ) = { h ( ` ) i } Ni=1 is the ` th layer node embedding matrix , h ( ` ) i ∈ Rd ` is the embedding of ith node , W ( ` ) ∈ Rd ` ×d ` −1 is the ` th layer weight matrix , and σ ( · ) is the ReLU activation . • ResGCN [ 33 ] solves the vanishing gradient issue by adding skip-connections between adjacency layers . More specifically , ResGCN computes node embeddings by H ( ` ) = σ ( PH ( ` −1 ) W ( ` ) ) + H ( ` −1 ) , where node embeddings of the previous layer is added to the output of the current layer to facilitate the training of deeper GCN models . • APPNP [ 29 ] adds skip-connections from the input layer to each hidden layer to preserve the feature information . APPNP computes node embeddings by H ( ` ) = α ` PH ( ` −1 ) + ( 1−α ` ) H ( 0 ) W , where α ` ∈ [ 0 , 1 ] balances the amount of information preserved at each layer . By decoupling feature transformation and propagation , APPNP can aggregate information from multi-hop neighbors without significantly increasing the computation complexity . • GCNII [ 6 ] improves the capacity of APPNP by adding non-linearty and weight matrix at each individual layer . GCNII computes node embeddings by H ( ` ) = σ ( ( α ` PH ( ` −1 ) + ( 1 − α ` ) H ( 0 ) ) W̄ ( ` ) ) , where W̄ ( ` ) = β ` W ( ` ) + ( 1− β ` ) I , constant α ` same as APPNP , and constant β ` ∈ [ 0 , 1 ] restricts the power of ` th layer parameters .

Summary: This paper analyzed GCN model for node classification in depth. The paper first analyzed the deep model's performance degeneration problem and argued that the deep model's expressiveness is not worse than shadow model so it does not suffer from oversmoothing problem. In fact, the degeneration of deep model comes from the hardness of training instead of oversmoothing, and the papers theoretically proved that deep model needs more iterations to converge, under a strong assumption. The above explanation explained the training loss but not characterize the final test performance. The author further explored the generalization error (the gap between training loss and test loss) and proved that under transductive setting the generalization error is affected by the number of training epochs, the number of layers, singluar value of weight matrix and largest node degrees, which explained why deep models have degenerated test performance. The author finally proposed a weight to decouple weight matrix and feature propagation operations, which keeps the expressiveness and also improves generalization ability.