AdaGCN: Adaboosting Graph Convolutional Networks into Deep Models

1 INTRODUCTION . Recently , research related to learning on graph structural data has gained considerable attention in machine learning community . Graph neural networks ( Gori et al. , 2005 ; Hamilton et al. , 2017 ; Veličković et al. , 2018 ) , particularly graph convolutional networks ( Kipf & Welling , 2017 ; Defferrard et al. , 2016 ; Bruna et al. , 2014 ) have demonstrated their remarkable ability on node classification ( Kipf & Welling , 2017 ) , link prediction ( Zhu et al. , 2016 ) and clustering tasks ( Fortunato , 2010 ) . Despite their enormous success , almost all of these models have shallow model architectures with only two or three layers . The shallow design of GCN appears counterintuitive as deep versions of these models , in principle , have access to more information , but perform worse . Oversmoothing ( Li et al. , 2018 ) has been proposed to explain why deep GCN fails , showing that by repeatedly applying Laplacian smoothing , GCN may mix the node features from different clusters and makes them indistinguishable . This also indicates that by stacking too many graph convolutional layers , the embedding of each node in GCN is inclined to converge to certain value ( Li et al. , 2018 ) , making it harder for classification . These shallow model architectures restricted by oversmoothing issue ∗Corresponding author . 1Code is available at https : //github.com/datake/AdaGCN . limit their ability to extract the knowledge from high-order neighbors , i.e. , features from remote hops of neighbors for current nodes . Therefore , it is crucial to design deep graph models such that high-order information can be aggregated in an effective way for better predictions . There are some works ( Xu et al. , 2018b ; Liao et al. , 2019 ; Klicpera et al. , 2018 ; Li et al. , 2019 ; Liu et al. , 2020 ) that tried to address this issue partially , and the discussion can refer to Appendix A.1 . By contrast , we argue that a key direction of constructing deep graph models lies in the efficient exploration and effective combination of information from different orders of neighbors . Due to the apparent sequential relationship between different orders of neighbors , it is a natural choice to incorporate boosting algorithm into the design of deep graph models . As an important realization of boosting theory , AdaBoost ( Freund et al. , 1999 ) is extremely easy to implement and keeps competitive in terms of both practical performance and computational cost ( Hastie et al. , 2009 ) . Moreover , boosting theory has been used to analyze the success of ResNets in computer vision ( Huang et al. , 2018 ) and AdaGAN ( Tolstikhin et al. , 2017 ) has already successfully incorporated boosting algorithm into the training of GAN ( Goodfellow et al. , 2014 ) . In this work , we focus on incorporating AdaBoost into the design of deep graph convolutional networks in a non-trivial way . Firstly , in pursuit of the introduction of AdaBoost framework , we refine the type of graph convolutions and thus obtain a novel RNN-like GCN architecture called AdaGCN . Our approach can efficiently extract knowledge from different orders of neighbors and then combine these information in an AdaBoost manner with iterative updating of the node weights . Also , we compare our AdaGCN with existing methods from the perspective of both architectural difference and feature representation power to show the benefits of our method . Finally , we conduct extensive experiments to demonstrate the consistent state-of-the-art performance of our approach across different label rates and computational advantage over other alternatives . 2 OUR APPROACH : ADAGCN . 2.1 ESTABLISHMENT OF ADAGCN . Consider an undirected graph G = ( V , E ) with N nodes vi ∈ V , edges ( vi , vj ) ∈ E . A ∈ RN×N is the adjacency matrix with corresponding degree matrix Dii = ∑ j Aij . In the vanilla GCN model ( Kipf & Welling , 2017 ) for semi-supervised node classification , the graph embedding of nodes with two convolutional layers is formulated as : Z = Â ReLU ( ÂXW ( 0 ) ) W ( 1 ) ( 1 ) where Z ∈ RN×K is the final embedding matrix ( output logits ) of nodes before softmax and K is the number of classes . X ∈ RN×C denotes the feature matrix where C is the input dimension . Â = D̃− 1 2 ÃD̃− 1 2 where Ã = A + I and D̃ is the degree matrix of Ã . In addition , W ( 0 ) ∈ RC×H is the input-to-hidden weight matrix for a hidden layer with H feature maps and W ( 1 ) ∈ RH×K is the hidden-to-output weight matrix . Our key motivation of constructing deep graph models is to efficiently explore information of highorder neighbors and then combine these messages from different orders of neighbors in an AdaBoost way . Nevertheless , if we naively extract information from high-order neighbors based on GCN , we are faced with stacking l layers ’ parameter matrix W ( i ) , i = 0 , ... , l − 1 , which is definitely costly in computation . Besides , Multi-Scale Deep Graph Convolutional Networks ( Luan et al. , 2019 ) also theoretically demonstrated that the output can only contain the stationary information of graph structure and loses all the local information in nodes for being smoothed if we simply deepen GCN . Intuitively , the desirable representation of node features does not necessarily need too many nonlinear transformation f applied on them . This is simply due to the fact that the feature of each node is normally one-dimensional sparse vector rather than multi-dimensional data structures , e.g. , images , that intuitively need deep convolution network to extract high-level representation for vision tasks . This insight has been empirically demonstrated in many recent works ( Wu et al. , 2019 ; Klicpera et al. , 2018 ; Xu et al. , 2018a ) , showing that a two-layer fully-connected neural networks is a better choice in the implementation . Similarly , our AdaGCN also follows this direction by choosing an appropriate f in each layer rather than directly deepen GCN layers . Thus , we propose to remove ReLU to avoid the expensive joint optimization of multiple parameter matrices . Similarly , Simplified Graph Convolution ( SGC ) ( Wu et al. , 2019 ) also adopted this prac- tice , arguing that nonlinearity between GCN layers is not crucial and the majority of the benefits arises from local weighting of neighboring features . Then the simplified graph convolution is : Z = ÂlXW ( 0 ) W ( 1 ) · · ·W ( l−1 ) = ÂlXW̃ , ( 2 ) where we collapse W ( 0 ) W ( 1 ) · · ·W ( l−1 ) as W̃ and Âl denotes Â to the l-th power . In particular , one crucial impact of ReLU in GCN is to accelerate the convergence of matrix multiplication since the ReLU is a contraction mapping intuitively . Thus , the removal of ReLU operation could also alleviate the oversmoothing issue , i.e . slowering the convergence of node embedding to indistinguishable ones ( Li et al. , 2018 ) . Additionally , without ReLU this simplified graph convolution is also able to avoid the aforementioned joint optimization over multiple parameter matrices , resulting in computational benefits . Nevertheless , we find that this type of stacked linear transformation from graph convolution has insufficient power in representing information of high-order neighbors , which is revealed in our experiment described in Appendix A.2 . Therefore , we propose to utilize an appropriate nonlinear function fθ , e.g. , a two-layer fully-connected neural network , to replace the linear transformation W̃ in Eq . 2 and enhance the representation ability of each base classifier in AdaGCN as follows : Z ( l ) = fθ ( Â lX ) , ( 3 ) where Z ( l ) represents the final embedding matrix ( output logits before Softmax ) after the l-th base classifier in AdaGCN . This formulation also implies that the l-th base classifier in AdaGCN is extracting knowledge from features of current nodes and their l-th hop of neighbors . Due to the fact that the function of l-th base classifier in AdaGCN is similar to that of the l-th layer in other traditional GCN-based methods that directly stack many graph convolutional layers , we regard the whole part of l-th base classifier as the l-th layers in AdaGCN . As for the realization of Multi-class AdaBoost , we apply SAMME ( Stagewise Additive Modeling using a Multi-class Exponential loss function ) algorithm ( Hastie et al. , 2009 ) , a natural and clean multi-class extension of the two-class AdaBoost adaptively combining weak classifiers . As illustrated in Figure 1 , we apply base classifier f ( l ) θ to extract knowledge from current node feature and l-th hop of neighbors by minimizing current weighted loss . Then we directly compute the weighted error rate err ( l ) and corresponding weight α ( l ) of current base classifier f ( l ) θ as follows : err ( l ) = n∑ i=1 wiI ( ci 6= f ( l ) θ ( xi ) ) / n∑ i=1 wi α ( l ) = log 1− err ( l ) err ( l ) + log ( K − 1 ) , ( 4 ) where wi denotes the weight of i-th node and ci represents the category of current i-th node . To attain a positive α ( l ) , we only need ( 1 − err ( l ) ) > 1/K , i.e. , the accuracy of each weak classifier should be better than random guess ( Hastie et al. , 2009 ) . This can be met easily to guarantee the weights to be updated in the right direction . Then we adjust nodes ’ weights by increasing weights on incorrectly classified ones : wi ← wi · exp ( α ( l ) · I ( ci 6= f ( l ) θ ( xi ) ) ) , i = 1 , . . . , n ( 5 ) After re-normalizing the weights , we then compute Âl+1X = Â · ( ÂlX ) to sequentially extract knowledge from l+1-th hop of neighbors in the following base classifier f ( l+1 ) θ . One crucial point of AdaGCN is that different from traditional AdaBoost , we only define one fθ , e.g . a two-layer fully connected neural network , which in practice is recursively optimized in each base classifier just similar to a recurrent neural network . This also indicates that the parameters from last base classifier are leveraged as the initialization of next base classifier , which coincides with our intuition that l+1-th hop of neighbors are directly connected from l-th hop of neighbors . The efficacy of this kind of layer-wise training has been similarly verified in ( Belilovsky et al. , 2018 ) recently . Further , we combine the predictions from different orders of neighbors in an Adaboost way to obtain the final prediction C ( A , X ) : C ( A , X ) = argmax k L∑ l=0 α ( l ) f ( l ) θ ( Â lX ) ( 6 ) Finally , we obtain the concise form of AdaGCN in the following : ÂlX = Â · ( Âl−1X ) Z ( l ) = f ( l ) θ ( Â lX ) Z = AdaBoost ( Z ( l ) ) ( 7 ) Note that fθ is non-linear , rather than linear in SGC ( Wu et al. , 2019 ) , to guarantee the representation power . As shown in Figure 1 , the architecture of AdaGCN is a variant of RNN with synchronous sequence input and output . Although the same classifier architecture is adopted for f ( l ) θ , their parameters are different , which is different from vanilla RNN . We provide a detailed description of the our algorithm in Section 3 .

Summary: By integrating Adaboosting and a fully connected layer, this paper provides a new graph neural network structure. The objective of this paper is to design a deeper graph models in an efficient way for better performance. The computational efficiency and performance of the proposed algorithm are evaluated using the task of node property prediction on several public datasets. This is a new variant of GNN, but the quality this paper is lower than the expectation regarding to the clarity and organisation.