Revisiting Layer-wise Sampling in Fast Training for Graph Convolutional Networks

To accelerate the training of graph convolutional networks ( GCNs ) , many sampling-based methods have been developed for approximating the embedding aggregation . Among them , a layer-wise approach recursively performs importance sampling to select neighbors jointly for existing nodes in each layer . This paper revisits the approach from a matrix approximation perspective . We identify two issues in the existing layer-wise sampling methods : sub-optimal sampling probabilities and the approximation bias induced by sampling without replacement . We thus propose remedies to address these issues . The improvements are demonstrated by extensive analyses and experiments on common benchmarks . 1 INTRODUCTION . Graph Convolutional Networks ( Kipf & Welling , 2017 ) are popular methods for learning the representation of nodes . However , it is computationally challenging to train a GCN over large-scale graphs due to the inter-dependence of nodes in a graph . In the mini-batch training for an L-layer GCN , the computation of embeddings involves not only the batch nodes but also batch nodes ’ Lhop neighbors , which is known as the phenomenon of “ neighbor explosion ” ( Zeng et al. , 2019 ) or “ neighbor expansion ” ( Chen et al. , 2018a ; Huang et al. , 2018 ) . To alleviate such a computation issue for large graphs , sampling-based methods are proposed to accelerate the training and reduce the memory cost . These approaches can be categorized as node-wise sampling approaches ( Hamilton et al. , 2017 ; Chen et al. , 2018a ) , subgraph sampling approaches ( Zeng et al. , 2019 ; Chiang et al. , 2019 ; Cong et al. , 2020 ) , and layer-wise sampling approaches ( Chen et al. , 2018b ; Huang et al. , 2018 ; Zou et al. , 2019 ) . We focus on layer-wise sampling in this work , which enjoys the efficiency and variance reduction by sampling columns of renormalized Laplacian matrix in each layer . This paper is a study of the existing sampling schemes in layer-wise sampling methods . We identify two potential drawbacks in the common practice for layer-wise sampling ( Chen et al. , 2018b ; Zou et al. , 2019 ) . First , the current sampling probabilities used are sub-optimal since a core assumption in FastGCN and LADIES does not hold in many common graph benchmarks , such as Reddit ( Hamilton et al. , 2017 ) and OGB ( Hu et al. , 2020 ) . Secondly , the previous implementations of the layer-wise sampling methods slightly deviate from their theoretical results , and introduce bias in the estimation due to the usage of sampling without replacement . Realizing the two issues , we accordingly propose the remedies with new sampling probabilities and a debiasing algorithm . The improvements of the proposed methods are demonstrated by extensive experiments on evaluating the matrix approximation error and prediction accuracy on large-scale benchmarks along with some theoretical analyses . To the best of our knowledge , our result is the first to recognize and resolve the issues with the default assumption and the practical implementation of layer-wise sampling methods for GCN . Once these sub-optimal practices are addressed , we observe that the GCN models consistently converge faster in training and usually enjoy a higher prediction accuracy . We believe the proposed methods can more generally improve the training for GCN as well , e.g. , the same strategy can allow node-wise sampling methods to adopt sampling without replacement and further improve the approximation accuracy . Moreover , our discussion on the bias induced by sampling without replacement is not limited to GCN , and the debiasing algorithm we develop can contribute to other sampling-based machine learning models beyond layer-wise sampling . 1.1 BACKGROUND AND RELATED WORK . GCN Graph Convolutional Networks ( GCNs , Kipf & Welling ( 2017 ) ) effectively incorporate the technique of convolution filter into the graph domain ( Wu et al. , 2020 ; Bronstein et al. , 2017 ) . Viewed as an approximation for the spectral graph convolutions ( Bruna et al. , 2014 ; Defferrard et al. , 2016 ) , GCN has achieved great success in learning tasks such as node classification and link prediction , with applications ranging from recommender systems ( Ying et al. , 2018 ) , traffic prediction ( Cui et al. , 2019 ; Rahimi et al. , 2018 ) , and knowledge graphs ( Schlichtkrull et al. , 2018 ) . Sampling Based GCN Training To name a few of sampling schemes , GraphSAGE ( Hamilton et al. , 2017 ) first introduces the “ node-wise ” neighbor sampling scheme , where a fixed number of neighbors are uniformly and independently sampled for each node in every layer . To reduce variance in node-wise sampling training , VR-GCN ( Chen et al. , 2018a ) applies a control variate approach with historical activation . Instead of sampling for each node , “ layer-wise ” sampling is a more efficient approach : joint sampling scheme for all the existing nodes in each layer so that these nodes can share the sampled neighboring node . FastGCN ( Chen et al. , 2018b ) first introduces this scheme with importance sampling . AS-GCN ( Huang et al. , 2018 ) proposes an alternative sampling probability for layer-wise sampling by approximating the hidden layer in the sampling procedure . Then Zou et al . ( 2019 ) propose a layer-dependent importance sampling scheme ( LADIES ) to further reduce the variance in training . This alleviates the issue of empty rows in sampled adjacency matrix for FastGCN . For the “ subgraph ” approach , ClusterGCN ( Chiang et al. , 2019 ) samples a dense subgraph associated with the batch nodes by graph clustering algorithm ; GraphSAINT ( Zeng et al. , 2019 ) introduces normalization and variance reduction in subgraph sampling . 2 NOTATIONS AND PRELIMINARIES . 2.1 GRAPH CONVOLUTIONAL NETWORKS . The GCN architecture for semi-supervised node classification is introduced by Kipf & Welling ( 2017 ) . Suppose we have an undirected graph G = ( V , E ) , where V is the set of n nodes and E is the set of E edges . Denote node i in V as vi , where i ∈ [ n ] is the index of nodes in the graph and [ n ] denotes the set { 1 , 2 , ... , n } . Each node vi ∈ V is associated with a feature vector xi ∈ Rp and a label vector yi ∈ Rq . Though we can observe the feature of every node in V and every edge in E , i.e . the n × n adjacency matrix A , we are only able to observe the label of partial nodes Vtrain , satisfying Vtrain ⊂ V . Thus , we need to predict the labels for the rest nodes in V\Vtrain and it becomes a semi-supervised learning task . A graph convolution layer is defined as : Z ( l+1 ) = PH ( l ) W ( l ) , H ( l ) = σ ( Z ( l ) ) , ( 1 ) where σ is an activation function and P is obtained from applying normalization to the graph adjacency matrix A ; H ( l ) is the embedding matrix of the graph nodes in the l-th layer , and W ( l ) is the parameter matrix of the same layer . In particular , H ( 0 ) is the n × p feature matrix . For mini-batch training , the training loss for an L-layer GCN is defined as 1|Vbatch| ∑ vi∈Vbatch ` ( yi , z ( L ) i ) , where ` is the loss function , batch nodes Vbatch is a subset of Vtrain at each iteration . z ( L ) i is the i-th row in Z ( L ) , | · | denotes the cardinality of a set . In this paper , we set P = D̃−1/2 ( A + I ) D̃−1/2 , where D̃ is a diagonal matrix with Dii = 1 + ∑ iAij . The matrix P is constructed as a renormalized Laplacian matrix to help alleviate overfitting and exploding/vanishing gradients issues ( Kipf & Welling , 2017 ) , which is used by Kipf & Welling ( 2017 ) ; Chen et al . ( 2018a ) ; Cong et al . ( 2020 ) . 2.2 LAYER-WISE SAMPLING . To address “ neighbor explosion ” issue for graph neural networks , sampling methods are integrated into the stochastic training . Motivated by the idea to approximate the matrix PH ( l ) in ( 1 ) , FastGCN ( Chen et al. , 2018b ) applies an importance-sampling-based strategy . Instead of individually sampling neighbors for each node in the l-th layer , they sample a set of s neighbors S ( l ) from V with importance sampling probability pi , where pi ∝ ∑n j=1P 2 ji and ∑ i pi = 1 . For the ( l− 1 ) -th layer , they naturally set V ( l−1 ) = S ( l ) . LADIES ( Zou et al. , 2019 ) improves the importance sampling probability pi as p ( l ) i ∝ ∑ vj∈N ( l ) P 2 ji , ∀i ∈ [ n ] ( 2 ) where N ( l ) = ∪vi∈V ( l ) N ( vi ) and ∑ j p ( l ) j = 1 . In this case , S ( l ) the nodes sampled for the l-th layer are guaranteed to be within the neighborhood of V ( l ) . The whole procedure can be concluded by a diagonal matrix S ( l ) ∈ Rn×n and a row selection matrixQ ( l ) ∈ Rsl×n , which are defined as Q ( l ) k , j = { 1 , j = i ( l ) k 0 , else , S ( l ) j , j = { ( slp ( l ) i ( l ) k ) −1 , j = i ( l ) k 0 , else , where { i ( l ) k } sl k=1 are the indices of rows selected in the l-th layer . The forward propagation with layer-wise sampling can thus be equivalently represented as Z̃ ( l+1 ) = Q ( l+1 ) PS ( l ) H ( l ) W ( l ) , H ( l ) = ( Q ( l ) ) Tσ ( Z̃ ( l ) ) , where Z̃ ( l+1 ) is the approximation of the embedding matrix for layer l . 3 EXPERIMENTAL SETUP . In advance of the formal introduction to the the issues and the corresponding remedies in Section 4 and Section 5 , we state the basic setups of the main experiments and datasets as they appear multiple times across the paper . Details about GCN model training are deferred to the according sections . Main experiments . To study the influence of the aforementioned issues we evaluate the matrix approximation error ( c.f . Figure 1 ) of different methods and consider it as a new metric to reflect the performance of the sampling strategy on approximating the original mini-batch training in one-step propagation . Since the updates of parameters in the training is not involved in the simple metric above , in Section 6 we further evaluate the prediction accuracy on testing sets of both intermediate models during training and final outputs , using the metrics in Table 2 . Benchmarks . Empirical experiments are conducted on 5 datasets ( see details at Table 2 in Appendix B ) : Reddit ( Hamilton et al. , 2017 ) , ogbn-arxiv , ogbn-proteins , ogbn-mag and ogbn-products ( Hu et al. , 2020 ) . Reddit is a traditional large graph dataset used by Chen et al . ( 2018b ) ; Zou et al . ( 2019 ) ; Chen et al . ( 2018a ) ; Cong et al . ( 2020 ) ; Zeng et al . ( 2019 ) . Ogbn-arxiv , ogbn-proteins and ogbn-products are Open Graph Benchmarks ( OGB ) proposed by Hu et al . ( 2020 ) . Compared to traditional datasets , our selected OGB data have larger volume ( up to million-node scale ) with more challenging data split . The metrics in Table 2 follow the choices of recent works and the recommendation by ( Hu et al. , 2020 ) .

Summary: This paper studies existing sampling schemes employed in training graph neural network architectures (e.g., FastGCN [Chen et al.] and LADIES [Zon et al.]) that are improvements to graph convolutional networks (GCN) [Kipf & Welling 2017] from a matrix approximation perspective (looking a Frobenius norm).  The paper focuses on layer-wise sampling (cf. node and subgraph sampling) and observes that there are two drawbacks in common practice layer-wise sampling. First drawback is current probabilities (probability distributions) used for sampling are sub-optimal. The reason the paper gives is that a core assumption made by certain GCN schemes such as FastGCN and LADIES do not hold in datasets such as Reddit and OGB. Second drawback is the implementations of these schemes slightly deviate from their respective theoretical results. The implementations use sampling without replacement, and thus introduces bias.  To this, the paper presents new sampling probabilities (uniform proposal distribution), and an algorithm to reduce the bias. With these adjustments, training of GCN converges faster and potentially leads to higher prediction accuracy (for node prediction). Theoretical analysis is presented and experiments are conducted on common benchmarks (Reddit, ogbn-{arxiv,proteins,products,mag}).