Revisiting Layer-wise Sampling in Fast Training for Graph Convolutional Networks

To accelerate the training of graph convolutional networks ( GCNs ) , many sampling-based methods have been developed for approximating the embedding aggregation . Among them , a layer-wise approach recursively performs importance sampling to select neighbors jointly for existing nodes in each layer . This paper revisits the approach from a matrix approximation perspective . We identify two issues in the existing layer-wise sampling methods : sub-optimal sampling probabilities and the approximation bias induced by sampling without replacement . We thus propose remedies to address these issues . The improvements are demonstrated by extensive analyses and experiments on common benchmarks . 1 INTRODUCTION . Graph Convolutional Networks ( Kipf & Welling , 2017 ) are popular methods for learning the representation of nodes . However , it is computationally challenging to train a GCN over large-scale graphs due to the inter-dependence of nodes in a graph . In the mini-batch training for an L-layer GCN , the computation of embeddings involves not only the batch nodes but also batch nodes ’ Lhop neighbors , which is known as the phenomenon of “ neighbor explosion ” ( Zeng et al. , 2019 ) or “ neighbor expansion ” ( Chen et al. , 2018a ; Huang et al. , 2018 ) . To alleviate such a computation issue for large graphs , sampling-based methods are proposed to accelerate the training and reduce the memory cost . These approaches can be categorized as node-wise sampling approaches ( Hamilton et al. , 2017 ; Chen et al. , 2018a ) , subgraph sampling approaches ( Zeng et al. , 2019 ; Chiang et al. , 2019 ; Cong et al. , 2020 ) , and layer-wise sampling approaches ( Chen et al. , 2018b ; Huang et al. , 2018 ; Zou et al. , 2019 ) . We focus on layer-wise sampling in this work , which enjoys the efficiency and variance reduction by sampling columns of renormalized Laplacian matrix in each layer . This paper is a study of the existing sampling schemes in layer-wise sampling methods . We identify two potential drawbacks in the common practice for layer-wise sampling ( Chen et al. , 2018b ; Zou et al. , 2019 ) . First , the current sampling probabilities used are sub-optimal since a core assumption in FastGCN and LADIES does not hold in many common graph benchmarks , such as Reddit ( Hamilton et al. , 2017 ) and OGB ( Hu et al. , 2020 ) . Secondly , the previous implementations of the layer-wise sampling methods slightly deviate from their theoretical results , and introduce bias in the estimation due to the usage of sampling without replacement . Realizing the two issues , we accordingly propose the remedies with new sampling probabilities and a debiasing algorithm . The improvements of the proposed methods are demonstrated by extensive experiments on evaluating the matrix approximation error and prediction accuracy on large-scale benchmarks along with some theoretical analyses . To the best of our knowledge , our result is the first to recognize and resolve the issues with the default assumption and the practical implementation of layer-wise sampling methods for GCN . Once these sub-optimal practices are addressed , we observe that the GCN models consistently converge faster in training and usually enjoy a higher prediction accuracy . We believe the proposed methods can more generally improve the training for GCN as well , e.g. , the same strategy can allow node-wise sampling methods to adopt sampling without replacement and further improve the approximation accuracy . Moreover , our discussion on the bias induced by sampling without replacement is not limited to GCN , and the debiasing algorithm we develop can contribute to other sampling-based machine learning models beyond layer-wise sampling . 1.1 BACKGROUND AND RELATED WORK . GCN Graph Convolutional Networks ( GCNs , Kipf & Welling ( 2017 ) ) effectively incorporate the technique of convolution filter into the graph domain ( Wu et al. , 2020 ; Bronstein et al. , 2017 ) . Viewed as an approximation for the spectral graph convolutions ( Bruna et al. , 2014 ; Defferrard et al. , 2016 ) , GCN has achieved great success in learning tasks such as node classification and link prediction , with applications ranging from recommender systems ( Ying et al. , 2018 ) , traffic prediction ( Cui et al. , 2019 ; Rahimi et al. , 2018 ) , and knowledge graphs ( Schlichtkrull et al. , 2018 ) . Sampling Based GCN Training To name a few of sampling schemes , GraphSAGE ( Hamilton et al. , 2017 ) first introduces the “ node-wise ” neighbor sampling scheme , where a fixed number of neighbors are uniformly and independently sampled for each node in every layer . To reduce variance in node-wise sampling training , VR-GCN ( Chen et al. , 2018a ) applies a control variate approach with historical activation . Instead of sampling for each node , “ layer-wise ” sampling is a more efficient approach : joint sampling scheme for all the existing nodes in each layer so that these nodes can share the sampled neighboring node . FastGCN ( Chen et al. , 2018b ) first introduces this scheme with importance sampling . AS-GCN ( Huang et al. , 2018 ) proposes an alternative sampling probability for layer-wise sampling by approximating the hidden layer in the sampling procedure . Then Zou et al . ( 2019 ) propose a layer-dependent importance sampling scheme ( LADIES ) to further reduce the variance in training . This alleviates the issue of empty rows in sampled adjacency matrix for FastGCN . For the “ subgraph ” approach , ClusterGCN ( Chiang et al. , 2019 ) samples a dense subgraph associated with the batch nodes by graph clustering algorithm ; GraphSAINT ( Zeng et al. , 2019 ) introduces normalization and variance reduction in subgraph sampling . 2 NOTATIONS AND PRELIMINARIES . 2.1 GRAPH CONVOLUTIONAL NETWORKS . The GCN architecture for semi-supervised node classification is introduced by Kipf & Welling ( 2017 ) . Suppose we have an undirected graph G = ( V , E ) , where V is the set of n nodes and E is the set of E edges . Denote node i in V as vi , where i ∈ [ n ] is the index of nodes in the graph and [ n ] denotes the set { 1 , 2 , ... , n } . Each node vi ∈ V is associated with a feature vector xi ∈ Rp and a label vector yi ∈ Rq . Though we can observe the feature of every node in V and every edge in E , i.e . the n × n adjacency matrix A , we are only able to observe the label of partial nodes Vtrain , satisfying Vtrain ⊂ V . Thus , we need to predict the labels for the rest nodes in V\Vtrain and it becomes a semi-supervised learning task . A graph convolution layer is defined as : Z ( l+1 ) = PH ( l ) W ( l ) , H ( l ) = σ ( Z ( l ) ) , ( 1 ) where σ is an activation function and P is obtained from applying normalization to the graph adjacency matrix A ; H ( l ) is the embedding matrix of the graph nodes in the l-th layer , and W ( l ) is the parameter matrix of the same layer . In particular , H ( 0 ) is the n × p feature matrix . For mini-batch training , the training loss for an L-layer GCN is defined as 1|Vbatch| ∑ vi∈Vbatch ` ( yi , z ( L ) i ) , where ` is the loss function , batch nodes Vbatch is a subset of Vtrain at each iteration . z ( L ) i is the i-th row in Z ( L ) , | · | denotes the cardinality of a set . In this paper , we set P = D̃−1/2 ( A + I ) D̃−1/2 , where D̃ is a diagonal matrix with Dii = 1 + ∑ iAij . The matrix P is constructed as a renormalized Laplacian matrix to help alleviate overfitting and exploding/vanishing gradients issues ( Kipf & Welling , 2017 ) , which is used by Kipf & Welling ( 2017 ) ; Chen et al . ( 2018a ) ; Cong et al . ( 2020 ) . 2.2 LAYER-WISE SAMPLING . To address “ neighbor explosion ” issue for graph neural networks , sampling methods are integrated into the stochastic training . Motivated by the idea to approximate the matrix PH ( l ) in ( 1 ) , FastGCN ( Chen et al. , 2018b ) applies an importance-sampling-based strategy . Instead of individually sampling neighbors for each node in the l-th layer , they sample a set of s neighbors S ( l ) from V with importance sampling probability pi , where pi ∝ ∑n j=1P 2 ji and ∑ i pi = 1 . For the ( l− 1 ) -th layer , they naturally set V ( l−1 ) = S ( l ) . LADIES ( Zou et al. , 2019 ) improves the importance sampling probability pi as p ( l ) i ∝ ∑ vj∈N ( l ) P 2 ji , ∀i ∈ [ n ] ( 2 ) where N ( l ) = ∪vi∈V ( l ) N ( vi ) and ∑ j p ( l ) j = 1 . In this case , S ( l ) the nodes sampled for the l-th layer are guaranteed to be within the neighborhood of V ( l ) . The whole procedure can be concluded by a diagonal matrix S ( l ) ∈ Rn×n and a row selection matrixQ ( l ) ∈ Rsl×n , which are defined as Q ( l ) k , j = { 1 , j = i ( l ) k 0 , else , S ( l ) j , j = { ( slp ( l ) i ( l ) k ) −1 , j = i ( l ) k 0 , else , where { i ( l ) k } sl k=1 are the indices of rows selected in the l-th layer . The forward propagation with layer-wise sampling can thus be equivalently represented as Z̃ ( l+1 ) = Q ( l+1 ) PS ( l ) H ( l ) W ( l ) , H ( l ) = ( Q ( l ) ) Tσ ( Z̃ ( l ) ) , where Z̃ ( l+1 ) is the approximation of the embedding matrix for layer l . 3 EXPERIMENTAL SETUP . In advance of the formal introduction to the the issues and the corresponding remedies in Section 4 and Section 5 , we state the basic setups of the main experiments and datasets as they appear multiple times across the paper . Details about GCN model training are deferred to the according sections . Main experiments . To study the influence of the aforementioned issues we evaluate the matrix approximation error ( c.f . Figure 1 ) of different methods and consider it as a new metric to reflect the performance of the sampling strategy on approximating the original mini-batch training in one-step propagation . Since the updates of parameters in the training is not involved in the simple metric above , in Section 6 we further evaluate the prediction accuracy on testing sets of both intermediate models during training and final outputs , using the metrics in Table 2 . Benchmarks . Empirical experiments are conducted on 5 datasets ( see details at Table 2 in Appendix B ) : Reddit ( Hamilton et al. , 2017 ) , ogbn-arxiv , ogbn-proteins , ogbn-mag and ogbn-products ( Hu et al. , 2020 ) . Reddit is a traditional large graph dataset used by Chen et al . ( 2018b ) ; Zou et al . ( 2019 ) ; Chen et al . ( 2018a ) ; Cong et al . ( 2020 ) ; Zeng et al . ( 2019 ) . Ogbn-arxiv , ogbn-proteins and ogbn-products are Open Graph Benchmarks ( OGB ) proposed by Hu et al . ( 2020 ) . Compared to traditional datasets , our selected OGB data have larger volume ( up to million-node scale ) with more challenging data split . The metrics in Table 2 follow the choices of recent works and the recommendation by ( Hu et al. , 2020 ) .

Summary: This paper revisits layer-wise sampling methods of graph neural networks by addressing issues such as sub-optimal sampling probabilities and the approximation bias due to the usage of sampling without replacement. This paper also presents a new metric to evaluate the performance of the sampling strategy. Through comprehensive experiments, the authors validate the effectiveness of their proposed methods.