GraphSAINT: Graph Sampling Based Inductive Learning Method

1 INTRODUCTION . Recently , representation learning on graphs has attracted much attention , since it greatly facilitates tasks such as classification and clustering ( Wu et al. , 2019 ; Cai et al. , 2017 ) . Current works on Graph Convolutional Networks ( GCNs ) ( Hamilton et al. , 2017 ; Chen et al. , 2018b ; Gao et al. , 2018 ; Huang et al. , 2018 ; Chen et al. , 2018a ) mostly focus on shallow models ( 2 layers ) on relatively small graphs . Scaling GCNs to larger datasets and deeper layers still requires fast alternate training methods . In a GCN , data to be gathered for one output node comes from its neighbors in the previous layer . Each of these neighbors in turn , gathers its output from the previous layer , and so on . Clearly , the deeper we back track , the more multi-hop neighbors to support the computation of the root . The number of support nodes ( and thus the training time ) potentially grows exponentially with the GCN depth . To mitigate such “ neighbor explosion ” , state-of-the-art methods use various layer sampling techniques . The works by Hamilton et al . ( 2017 ) ; Ying et al . ( 2018a ) ; Chen et al . ( 2018a ) ensure that only a small number of neighbors ( typically from 2 to 50 ) are selected by one node in the next layer . Chen et al . ( 2018b ) and Huang et al . ( 2018 ) further propose samplers to restrict the neighbor expansion factor to 1 , by ensuring a fixed sample size in all layers . While these methods significantly speed up training , they face challenges in scalability , accuracy or computation complexity . ∗Equal contribution Present work We present GraphSAINT ( Graph SAmpling based INductive learning meThod ) to efficiently train deep GCNs . GraphSAINT is developed from a fundamentally different way of minibatch construction . Instead of building a GCN on the full training graph and then sampling across the layers , we sample the training graph first and then build a full GCN on the subgraph . Our method is thus graph sampling based . Naturally , GraphSAINT resolves “ neighbor explosion ” , since every GCN of the minibatches is a small yet complete one . On the other hand , graph sampling based method also brings new challenges in training . Intuitively , nodes of higher influence on each other should have higher probability to form a subgraph . This enables the sampled nodes to “ support ” each other without going outside the minibatch . Unfortunately , such strategy results in non-identical node sampling probability , and introduces bias in the minibatch estimator . To address this issue , we develop normalization techniques so that the feature learning does not give preference to nodes more frequently sampled . To further improve training quality , we perform variance reduction analysis , and design light-weight sampling algorithms by quantifying “ influence ” of neighbors . Experiments on GraphSAINT using five large datasets show significant performance gain in both training accuracy and time . We also demonstrate the flexibility of GraphSAINT by integrating our minibatch method with popular GCN architectures such as JK-net ( Xu et al. , 2018 ) and GAT ( Veličković et al. , 2017 ) . The resulting deep models achieve new state-of-the-art F1 scores on PPI ( 0.995 ) and Reddit ( 0.970 ) . 2 RELATED WORK . A neural network model that extends convolution operation to the graph domain is first proposed by Bruna et al . ( 2013 ) . Further , Kipf & Welling ( 2016 ) ; Defferrard et al . ( 2016 ) speed up graph convolution computation with localized filters based on Chebyshev expansion . They target relatively small datasets and thus the training proceeds in full batch . In order to scale GCNs to large graphs , layer sampling techniques ( Hamilton et al. , 2017 ; Chen et al. , 2018b ; Ying et al. , 2018a ; Chen et al. , 2018a ; Gao et al. , 2018 ; Huang et al. , 2018 ) have been proposed for efficient minibatch training . All of them follow the three meta steps : 1 . Construct a complete GCN on the full training graph . 2 . Sample nodes or edges of each layer to form minibatches . 3 . Propagate forward and backward among the sampled GCN . Steps ( 2 ) and ( 3 ) proceed iteratively to update the weights via stochastic gradient descent . The layer sampling algorithm of GraphSAGE ( Hamilton et al. , 2017 ) performs uniform node sampling on the previous layer neighbors . It enforces a pre-defined budget on the sample size , so as to bound the minibatch computation complexity . Ying et al . ( 2018a ) enhances the layer sampler of Hamilton et al . ( 2017 ) by introducing an importance score to each neighbor . The algorithm presumably leads to less information loss due to weighted aggregation . S-GCN ( Chen et al. , 2018a ) further restricts neighborhood size by requiring only two support nodes in the previous layer . The idea is to use the historical activations in the previous layer to avoid redundant re-evaluation . FastGCN ( Chen et al. , 2018b ) performs sampling from another perspective . Instead of tracking down the inter-layer connections , node sampling is performed independently for each layer . It applies importance sampling to reduce variance , and results in constant sample size in all layers . However , the minibatches potentially become too sparse to achieve high accuracy . Huang et al . ( 2018 ) improves FastGCN by an additional sampling neural network . It ensures high accuracy , since sampling is conditioned on the selected nodes in the next layer . Significant overhead may be incurred due to the expensive sampling algorithm and the extra sampler parameters to be learned . Instead of sampling layers , the works of Zeng et al . ( 2018 ) and Chiang et al . ( 2019 ) build minibatches from subgraphs . Zeng et al . ( 2018 ) proposes a specific graph sampling algorithm to ensure connectivity among minibatch nodes . They further present techniques to scale such training on shared-memory multi-core platforms . More recently , ClusterGCN ( Chiang et al. , 2019 ) proposes graph clustering based minibatch training . During pre-processing , the training graph is partitioned into densely connected clusters . During training , clusters are randomly selected to form minibatches , and intra-cluster edge connections remain unchanged . Similar to GraphSAINT , the works of Zeng et al . ( 2018 ) and Chiang et al . ( 2019 ) do not sample the layers and thus “ neighbor explosion ” is avoided . Unlike GraphSAINT , both works are heuristic based , and do not account for bias due to the unequal probability of each node / edge appearing in a minibatch . Another line of research focuses on improving model capacity . Applying attention on graphs , the architectures of Veličković et al . ( 2017 ) ; Zhang et al . ( 2018 ) ; Lu et al . ( 2019 ) better capture neighbor features by dynamically adjusting edge weights . Klicpera et al . ( 2018 ) combines PageRank with GCNs to enable efficient information propagation from many hops away . To develop deeper models , “ skip-connection ” is borrowed from CNNs ( He et al. , 2015 ; Huang et al. , 2017 ) into the GCN context . In particular , JK-net Xu et al . ( 2018 ) demonstrates significant accuracy improvement on GCNs with more than two layers . Note , however , that JK-net ( Xu et al. , 2018 ) follows the same sampling strategy as GraphSAGE ( Hamilton et al. , 2017 ) . Thus , its training cost is high due to neighbor explosion . In addition , high order graph convolutional layers ( Zhou , 2017 ; Lee et al. , 2018 ; Abu-El-Haija et al. , 2019 ) also help propagate long-distance features . With the numerous architectural variants developed , the question of how to train them efficiently via minibatches still remains to be answered . 3 PROPOSED METHOD : GraphSAINT Graph sampling based method is motivated by the challenges in scalability ( in terms of model depth and graph size ) . We analyze the bias ( Section 3.2 ) and variance ( Section 3.3 ) introduced by graph sampling , and thus , propose feasible sampling algorithms ( Section 3.4 ) . We show the applicability of GraphSAINT to other architectures , both conceptually ( Section 4 ) and experimentally ( Section 5.2 ) . In the following , we define the problem of interest and the corresponding notations . A GCN learns representation of an un-directed , attributed graph G ( V , E ) , where each node v ∈ V has a length-f attribute xv . Let A be the adjacency matrix and Ã be the normalized one ( i.e. , Ã = D−1A , and D is the diagonal degree matrix ) . Let the dimension of layer- ` input activation be f ( ` ) . The activation of node v is x ( ` ) v ∈ Rf ( ` ) , and the weight matrix is W ( ` ) ∈ Rf ( ` ) ×f ( ` +1 ) . Note that xv = x ( 1 ) v . Propagation rule of a layer is defined as follows : x ( ` +1 ) v = σ ( ∑ u∈V Ãv , u ( W ( ` ) ) T x ( ` ) u ) ( 1 ) where Ãv , u is a scalar , taking an element of Ã . And σ ( · ) is the activation function ( e.g. , ReLU ) . We use subscript “ s ” to denote parameterd of the sampled graph ( e.g. , Gs , Vs , Es ) . GCNs can be applied under inductive and transductive settings . While GraphSAINT is applicable to both , in this paper , we focus on inductive learning . It has been shown that inductive learning is especially challenging ( Hamilton et al. , 2017 ) — during training , neither attributes nor connections of the test nodes are present . Thus , an inductive model has to generalize to completely unseen graphs . 3.1 MINIBATCH BY GRAPH SAMPLING GraphSAINT follows the design philosophy of directly sampling the training graph G , rather than the corresponding GCN . Our goals are to 1. extract appropriately connected subgraphs so that little information is lost when propagating within the subgraphs , and 2. combine information of many subgraphs together so that the training process overall learns good representation of the full graph . Figure 1 and Algorithm 1 illustrate the training algorithm . Before training starts , we perform light-weight pre-processing on G with the given sampler SAMPLE . The pre-processing estimates the probability of a node v ∈ V and an edge e ∈ E being sampled by SAMPLE . Such probability is later used to normalize the subgraph neighbor aggregation and the minibatch loss ( Section 3.2 ) . Afterwards , Algorithm 1 GraphSAINT training algorithm Input : Training graph G ( V , E , X ) ; Labels Y ; Sampler SAMPLE ; Output : GCN model with trained weights 1 : Pre-processing : Setup SAMPLE parameters ; Compute normalization coefficients α , λ . 2 : for each minibatch do 3 : Gs ( Vs , Es ) ← Sampled sub-graph of G according to SAMPLE 4 : GCN construction on Gs . 5 : { yv | v ∈ Vs } ← Forward propagation of { xv | v ∈ Vs } , normalized by α 6 : Backward propagation from λ-normalized loss L ( yv , yv ) . Update weights . 7 : end for training proceeds by iterative weight updates via SGD . Each iteration starts with an independently sampled Gs ( where |Vs| |V| ) . We then build a full GCN on Gs to generate embedding and calculate loss for every v ∈ Vs . In Algorithm 1 , node representation is learned by performing node classification in the supervised setting , and each training node v comes with a ground truth label yv . Intuitively , there are two requirements for SAMPLE : 1 . Nodes having high influence on each other should be sampled in the same subgraph . 2 . Each edge should have non-negligible probability to be sampled . For requirement ( 1 ) , an ideal SAMPLE would consider the joint information from node connections as well as attributes . However , the resulting algorithm may have high complexity as it would need to infer the relationships between features . For simplicity , we define “ influence ” from the graph connectivity perspective and design topology based samplers . Requirement ( 2 ) leads to better generalization since it enables the neural net to explore the full feature and label space .

Summary: This paper proposed a new sampling method to train GCN in the mini-batch manner. In particular, unlike existing methods which samples the mini-batch in the node-wise way, GraphSAINT proposed to sample a mini-batch in the graph-wise way. As a result, GraphSAINT uses the same graph across different GCN layers, while most existing methods use different graphs across different GCN layers.  In addition, the authors show that this sampling method is unbiased. Extensive experimental results have shown improvement over existing methods. Overall, this idea is interesting and well presented.