Local Clustering Graph Neural Networks

1 INTRODUCTION . Recent emergence of the Graph Neural Networks ( GNNs ) , exemplified by models like ChebyNet ( Defferrard et al. , 2016 ) , GCN ( Kipf & Welling , 2017 ) , GraphSAGE ( Hamilton et al. , 2017 ) , GAT ( Veličković et al. , 2018 ) , and GIN ( Xu et al. , 2019 ) , has drastically reshaped the landscape of the graph learning research . These methods generalize traditional deep learning algorithms to model graph-structured data by combining graph propagation and neural networks . Despite its conceptual simplicity , GNNs have reestablished the new state-of-the-art methods in various graph learning tasks , such as node classification , link prediction , and graph classification ( Hu et al. , 2020 ; Dwivedi et al. , 2020 ) , also served as key contributors to many real-world applications , such as recommendation system ( Ying et al. , 2018 ) , smart transportation ( Luo et al. , 2020 ) , visual question answering ( Teney et al. , 2017 ) and molecular de-novo design ( You et al. , 2018 ) . With the growth of real-world social and information networks ( Leskovec et al. , 2005 ) , there is an urgent need to scale GNNs to massive graphs . For example , the recommendation systems in Alibaba ( Zhu et al. , 2019 ) and Pinterest ( Ying et al. , 2018 ) require training and inferring GNNs on graphs with billions of edges . Building such large-scale GNNs , however , is a notoriously expensive process . For instance , the GNN models in Pinterest are trained on a 500GB machine with 16 Tesla K80 GPUs , and served on a Hadoop cluster with 378 d2.8xlarge Amazon AWS machines . Although one may think model parameters are the main contributors to the huge resource consumption of GNNs , previous work ( Ma et al. , 2019 ) suggests the main bottleneck actually comes from the entanglement between graph propagation and neural networks , which leads to a large and irregular computation graph for GNNs . This problem is further exacerbated by the small-world phenomenon ( Watts & Strogatz , 1998 ) , i.e. , even a relatively small number of graph propagation can involve full-graph computation . For example , in Facebook college graphs of John Hopkins ( Traud et al. , 2012 ) , the 2-hop neighbors of node 1 , as shown in Fig . 1a , covers 74.5 % of the whole graph . A common strategy to reduce the overhead of GNNs is to make the graph smaller but may bring side effects . For instance , graph sampling techniques , such as neighborhood sampling in GraphSAGE ( Hamilton et al. , 2017 ) , may lead to the high variance issue ( Chen et al. , 2018a ) . Alternatively , graph partition techniques , such as METIS ( Karypis & Kumar , 1998 ) that adopted by Cluster-GCN ( Chiang et al. , 2019 ) and AliGraph ( Zhu et al. , 2019 ) , essentially involves extra full- graph computation for preprocessing . Besides reducing graph size , a recent attempt ( Wu et al. , 2019 ) to scale up GNNs is to decouple graph propagation and neural networks . However , such simplification may sacrifice certain performances . In this work , we explore a theoretically guaranteed strategy , local clustering ( Spielman & Teng , 2013 ; Andersen et al. , 2006 ) , intending to design a lightweight , effective and scalable GNN framework . We establish a connection between GNNs and local clustering theory , showing that the graph propagation in GNNs ( i.e. , short random-walk ) has a high probability to be stuck at a local cluster ( a.k.a , community ) , and the escaping probability is proportional to the conductance of the local cluster . We propose Local Clustering Graph Neural Networks ( LCGNN ) , which utilizes local clustering algorithms to seek local and dense subgraphs ( e.g. , Fig . 1b ) for GNN training and inference . Different from full-batch and graph partition-based methods , LCGNN does not incur full-graph processing and can be conducted locally . Compared to various baselines , LCGNN achieves stateof-the-art results in four Open Graph Benchmark ( Hu et al. , 2020 ) datasets . Moreover , the locality nature of LCGNN allows it to scale to graphs with 100M nodes and 1B edges on a single GPU . The rest of the paper is organized as follows . Section 2 gives a brief background summary followed by a survey of related works in section 3 . In section 4 and section 5 , we establish the connection between GNNs and local clustering , and then describe our LCGNN framework . Section 6 presents the experimental results and ablation study . Finally , we concludes this work in section 7 . 2 BACKGROUND . In this section , we bring the necessary background about graph , graph convolutional networks ( GCN ) , ( lazy ) random walk on graphs , and graph conductance . Graph Notations The graph G = ( V , E , A ) consists of |V | = n nodes and |E| = m edges . A ∈ Rn×n+ is the adjacency matrix where its entry A ( i , j ) , if nonzero , denote there is an edge between node i and j with edge weight Aij . In this work , we assume the input graph is undirected and unweighted , and our analysis can be generalized to the weighted graph case easily . For undirected graph , the degree matrix D , diag ( d ( 1 ) , · · · , d ( n ) ) is a diagonal matrix where d ( i ) , ∑ jA ( i , j ) is the degree of node i . Moreover , each node inG is associated with a F -dimensional feature vector , denoted by xi ∈ RF . The entire feature matrix X ∈ Rn×F is the concatenation of node feature vectors . There are two matrices that play importance roles in the design and analysis of GCN ( Kipf & Welling , 2017 ) — the normalized graph Laplacian L , D−1/2AD−1/2 and the random walk transition probability matrix P , AD−1 . Note that the entry P ( i , j ) indicates the probability that the random walk goes from node j to node i. Graph Convolutional Networks ( GCN ) GCN ( Kipf & Welling , 2017 ) initializes the node representation as the input feature matrix H ( 0 ) ←X , and iteratively apply non-linear transformation and graph propagation on node representation : H ( k ) ← ReLU ( LH ( k−1 ) W ( k ) ) , where left multiplying H ( k−1 ) by normalized graph Laplacian L acts as the graph propagation , and right multiplying H ( k−1 ) by W as well as the ReLU ( Glorot et al. , 2011 ) activation acts as the non-linear transformation . For the node classification task , a K-layer GCN predicts node labels Y with a softmax classifier : Y ← softmax ( LH ( K−1 ) W ( K ) ) . Take a two-layer GCN ( K = 2 ) as a running exam- ple , the predicted node labels Y is defined as Y ← softmax ( LReLU ( LH ( 0 ) W ( 1 ) ) W ( 2 ) ) . Lazy Random Walk In practice , many GNNs add ( weighted ) self-loops to the graphs ( A ← A+αI , Kipf & Welling ( 2017 ) ; Xu et al . ( 2019 ) ) or create residual connections ( He et al. , 2016 ) in neural networks ( Li et al. , 2019 ; Dehmamy et al. , 2019 ) . Such techniques can be viewed as variants of lazy random walk on graphs — at every step , with probability 1/2 the walker stays at the current node ( through a self-loop ) and with probability 1/2 the walker travels to a neighbor . The transition matrix of a lazy random walk is M , ( I + AD−1 ) /2 . In this work , we mainly consider lazy random walk . Because it has several desired properties and it is consistent with the actual situation . Graph Conductance For an undirected unweighted graph G = ( V , E , A ) , the graph volume of any non-empty node set S ⊂ V is defined as vol ( S ) , ∑ i∈S d ( i ) , which measures the total number of edges incident from S. The conductance of a non-empty node set S ⊂ V is defined as Φ ( S ) , ∑ i∈S ∑ j∈V−S A ( i , j ) min ( vol ( S ) , vol ( V−S ) ) . Roughly speaking , conductance Φ ( S ) is the ratio of the number of edges across S and V − S to the number of edges incident from S , measuring the clusterability of a subset S. Low conductance indicates a good cluster because its internal connections are significantly richer than its external connections . Although it is NP-hard to minimize conductance ( Šı́ma & Schaeffer , 2006 ) , there have been theoretically-guaranteed approximation algorithms that identify clusters near a given node that satisfy a target conductance condition , such as Spielman & Teng ( 2013 ) ; Andersen et al . ( 2006 ) ; Chung ( 2007 ) . 3 RELATED WORK . The design of scalable GNNs has attracted wide attention from the machine learning community . We review related work from three perspectives : ( 1 ) full-batch GNNs with co-design of systems and algorithms ; ( 2 ) sampling-based GNNs ; ( 3 ) graph partition-based GNNs . Full-batch GNNs A full-batch GNN takes a whole graph as input for forward and backward . Consequently , its computational cost is proportional to the graph size . Earlier GNN models ( Kipf & Welling , 2017 ; Veličković et al. , 2018 ) evaluated on relatively small graphs , thus can be trained in a full-batch manner . Scaling full-batch GNNs to large graphs requires the co-design of ML systems and ML algorithms ( Jiang et al. , 2020 ; Zhang et al. , 2020 ; Ma et al. , 2019 ) . For example , NeuGraph ( Ma et al. , 2019 ) runs full-batch GNN models on a graph with 8.6M nodes and 231.6M edges on an eight-P100-GPU server . SGC ( Wu et al. , 2019 ) is another attempt at full-batch GNN . It simplifies GCN by conducting graph propagation and classification separately and efficiently . However , such simplification may sacrifice performance in some downstream tasks . GNNs based on Graph Sampling GraphSAGE ( Hamilton et al. , 2017 ) first proposed the idea of neighborhood sampling , and later it was applied in a real-world recommendation system by PinSAGE ( Ying et al. , 2018 ) . At each GNN layer , GraphSAGE computes node representation by first down-sampling its neighborhoods and then aggregating the sampled ones . As a randomized algorithm , Neighborhood Sampling was further improved by FastGCN ( Chen et al. , 2018b ) , Stochastic GCN ( Chen et al. , 2018a ) and Adaptive Sampling ( Huang et al. , 2018 ) for variance reduction . A recent work about sampling-based GNN is GraphSAINT ( Zeng et al. , 2020 ) , which samples subgraphs ( Leskovec & Faloutsos , 2006 ) and run full-batch GNN on sampled subgraphs . GNNs based on Graph Partition Cluster-GCN ( Chiang et al. , 2019 ) is the most related work to ours . Cluster-GCN adopts global graph partition algorithms , METIS ( Karypis & Kumar , 1998 ) , to partition the input graph into subgraphs , and run a GNN on each subgraph . A similar idea was also proposed in AliGraph ( Zhu et al. , 2019 ) . However , global graph partition algorithms involve additional whole graph computation . Moreover , global graph partition algorithms are vulnerable to dynamic and evolving graphs ( Xu et al. , 2014 ; Vaquero et al. , 2014 ) , with nodes and edges being constantly added and removed , which are very common in real-world applications .

Summary: In this paper, the authors study the connection between GNNs and local clustering, and find that short random-walks in GNNs have a high probability to be stuck at a local cluster. Based on this, they propose a light and scalable GNN learning framework called LCGNN, which first adopts the local clustering method PPR-Nibble to partition full graph into subgraphs, then use GNN modules on subgraphs for training and inference. The authors evaluate LCGNN on six OGB datasets, and the proposed approach outperforms the competitors on node classification and link prediction tasks.