Deep Graph Neural Networks with Shallow Subgraph Samplers

1 INTRODUCTION . Graph Neural Networks ( GNNs ) have now become the state-of-the-art models for graph mining ( Wu et al. , 2020 ; Hamilton et al. , 2017b ; Zhang et al. , 2019 ) , facilitating applications such as social recommendation ( Monti et al. , 2017 ; Ying et al. , 2018 ; Pal et al. , 2020 ) , knowledge understanding ( Schlichtkrull et al. , 2018 ; Park et al. , 2019 ; Zhang et al. , 2020 ) and drug discovery ( Stokes et al. , 2020 ; Lo et al. , 2018 ) . With the numerous architectures proposed ( Kipf & Welling , 2016 ; Hamilton et al. , 2017a ; Veličković et al. , 2018 ) , it still remains an open question how to effectively design deep GNNs . There are two fundamental obstacles that are intrinsic to the underlying graph structure : • Expressivity challenge : deep GNNs tend to oversmooth ( Li et al. , 2018 ) . They collapse embeddings of different nodes into a fixed low-dimensional subspace after repeated neighbor mixing . • Computation challenge : deep GNNs recursively expand the adjacent nodes along message passing edges . The neighborhood size may grow exponentially with model depth ( Chen et al. , 2017 ) . Due to oversmoothing , one of the most popular GNN architectures , Graph Convolutional Network ( GCN ) ( Kipf & Welling , 2016 ) , has been theoretically proven as incapable of scaling to deep layers ( Oono & Suzuki , 2020 ; Rong et al. , 2020 ; Huang et al. , 2020 ) . Remedies to overcome the GCN limitations are two-folded . From the neural architecture perspective , researchers are actively seeking for more expressive neighbor aggregation operations ( Veličković et al. , 2018 ; Hamilton et al. , 2017a ; Xu et al. , 2018a ) , or transferring design components ( such as residual connection ) from deep CNNs to GNNs ( Xu et al. , 2018b ; Li et al. , 2019 ; Huang et al. , 2018 ) . From the data perspective , various works ( Klicpera et al. , 2019a ; b ; Bojchevski et al. , 2020 ) revisit classic graph analytic algorithms to reconstruct a graph with nicer topological property . The two kinds of works can also be combined to jointly improve the quality of message passing in deep GNNs . All the above GNN variants take a “ global ” view on the input graph G ( V , E ) — i.e. , all nodes are considered as belonging to the same G , whose size can often be massive . To generate the node embedding , no matter how we modify the architecture and the graph structure , a deep enough GNN would always propagate the influence from the entire node set V into a single target node . Intuitively , for a large graph , most nodes in V barely provide any useful information to the target nodes . We thus regard such “ global view ” on G as one of the root causes for both the expressivity and computation challenges discussed above . In this work , for the node embedding task , we take an alternative “ local view ” and interpret the GNN input as V = ⋃ v∈V V [ v ] and E = ⋃ v∈V E [ v ] . In other words , each target node v belongs to some small graph G [ v ] capturing the characteristics of only the node v. The entire input graph G is observed as the union of all such local yet latent G [ v ] . Such simple global-tolocal switch of perspective enables us to address both the expressivity and computation challenges without resorting to alternative GNN architectures or reconstructing the graph . Present work : SHADOW-GNN . We propose a “ Deep GNN , shallow sampler ” design principle that helps improve the expressive power and inference efficiency of various GNN architectures . We break the conventional thinking that an L-layer ( deep ) GNN has to aggregate L-hop ( faraway ) neighbors . We argue that the GNN receptive field for a target node should be shallower than the GNN depth . In other words , an L-layer GNN should only operate on a small subgraph G [ v ] surrounding the target node v , where G [ v ] consists of ( part of ) the L0-hop neighborhood . The deep vs. shallow comparison is reflected by setting L0 < L. We name such a GNN on G [ v ] as a SHADOW-GNN . We justify our design principle from two aspects . Firstly , why do we need the neighborhood to be shallow ? As a motivating example , the average number of 4-hop neighbors for the ogbn-products graph ( Hu et al. , 2020 ) is 0.6M , corresponding to 25 % of the full graph size . Blindly encoding the 0.6M node features into a single embedding vector can create the “ information bottleneck ” ( Alon & Yahav , 2020 ) . The irrelevant information from the majority of the 0.6M nodes may also “ dilute ” the truly useful signals from a small set of close neighbors . A simple solution to the above issues is to manually create a shallow neighborhood by subgraph sampling . The second question regarding SHADOW-GNN is : why do we still need deep GNNs ? Using more number of layers than the number of hops means the same pair of nodes may exchange messages with each other multiple times . Intuitively , this helps the GNN better absorb the subgraph information . Theoretically , we prove that a GNN deeper than the hops of the subgraph can be more powerful than the 1-dimensional WeisfeilerLehman test ( Shervashidze et al. , 2011 ) . A shallow GNN , on the contrary , can not accurately learn certain simple functions such as unweighted mean of the shallow neighborhood features . Note that with GCN as the backbone , a SHADOW-GCN still performs signal smoothing in each layer . However , the important distinction is that a deep GCN smooths the full G regardless of the target node , while a SHADOW-GCN constructs a customized smoothing domain G [ v ] for each target v. The variance in those smoothing domains created by SHADOW-GCN encourages variances in the node embedding vectors . With such intuition , our analysis shows that SHADOW-GNN does not oversmooth . Finally , since the sizes of the shallow neighborhoods are independent of the GNN depth , the computation challenge due to neighbor explosion is automatically addressed . We propose various subgraph samplers for SHADOW-GNN , including the simplest k-hop sampler and a sampler based on personalized PageRank , to improve the inference accuracy and computation efficiency . By experiments on five standard benchmarks , our SHADOW-SAGE and SHADOW-GAT models achieve significant accuracy gains compared with the original GraphSAGE and GAT models . In the meantime , the inference cost is reduced by orders of magnitude . 2 RELATED WORK AND PRELIMINARIES . Deep GNNs . Recently , numerous GNN models ( Kipf & Welling , 2016 ; Defferrard et al. , 2016 ; Hamilton et al. , 2017a ; Veličković et al. , 2018 ; Xu et al. , 2018b ; a ) have been proposed . In general , the input to a GNN is the graph G , and the outputs are representation vectors for each node , capturing both the feature and structural information of the neighborhood . Most state-of-the-art GNNs use shallow models ( i.e. , 2 to 3 layers ) . As first proposed by Li et al . ( 2018 ) and further elaborated by Luan et al . ( 2019 ) ; Oono & Suzuki ( 2020 ) ; Zhao & Akoglu ( 2020 ) ; Huang et al . ( 2020 ) , one of the major challenges to deepen GNNs is the “ oversmoothing ” of node features — each layer aggregation pushes the neighbor features towards similar values . Repeated aggregation over many layers results in node features being averaged over the full graph . A deep GNN may thus generate indistinguishable embeddings for different nodes . Viewing oversmoothing as a limitation of the layer aggregation , researchers develop alternative architectures . AS-GCN ( Huang et al. , 2018 ) , DeepGCN ( Li et al. , 2019 ) and JK-net ( Xu et al. , 2018b ) use skip-connection across layers . MixHop ( Abu-El-Haija et al. , 2019 ) , Snowball ( Luan et al. , 2019 ) and DAGNN ( Liu et al. , 2020 ) enable multi-hop message passing within a single layer . GraphSAGE ( Hamilton et al. , 2017a ) and GCNII ( Ming Chen et al. , 2020 ) encourage self-to-self message passing which effectively form an implicit skip-connection . GIN ( Xu et al. , 2018a ) and DeeperGCN ( Li et al. , 2020a ) propose more expressive neighbor aggregation operations . All the above focus on architectural exploration , which is a research direction orthogonal to ours . We can construct the SHADOW version of these GNNs in a plug-and-play fashion . Lastly , DropEdge ( Rong et al. , 2020 ) and Bayesian-GDC ( Hasanzadeh et al. , 2020 ) propose regularization techniques by adapting dropout ( Srivastava et al. , 2014 ) to graphs . Such techniques are only applied during training , and so oversmoothing during inference may not be alleviated . Learning from structural information . Another line of research is to go beyond the layer-wise message passing and more explicitly utilize the graph structural information ( Wu et al. , 2019 ; Klicpera et al. , 2019a ; Bojchevski et al. , 2020 ; Liu et al. , 2020 ; Frasca et al. , 2020 ; You et al. , 2019 ; Li et al. , 2020b ) . In particular , APPNP ( Klicpera et al. , 2019a ) and PPRGo ( Bojchevski et al. , 2020 ) utilize the personalized PageRank ( Page et al. , 1999 ) algorithm to re-define neighbor connections — instead of propagating features along the ( noisy ) graph edges , any nodes of structural significance can directly propagate to the target node . Other related methods such as GDC ( Klicpera et al. , 2019b ) and AM-GCN ( Wang et al. , 2020 ) reconstructs the adjacency matrix in each GNN layer to short-cut important multi-hop neighbors . Note that all the above methods takes a global view on G and operate the neural networks on the full graph . On the other hand , the idea of using subgraph samples to improve the GNN efficiency has also been explored . For example , SEAL ( Zhang & Chen , 2018 ) extracts local k-hop enclosing subgraphs to perform link prediction . GraphSAINT ( Zeng et al. , 2020 ) propose random walk samplers to construct minibatches during training . Notations . We focus on the node classification task , although our design principle can be naturally extended to other tasks . Let G ( V , E , X ) be an undirected graph , with node set V , edge set E ⊆ V×V and node feature matrix X ∈ R|V|×d . The u-th row of X corresponds to the length-d feature of node u . Let A be the adjacency matrix of G where Au , v = 1 if edge ( u , v ) ∈ E and Au , v = 0 otherwise . Denote Ã as the adjacency matrix after symmetric normalization ( used by GCN ) , and Â as the one after random walk normalization ( used by GraphSAGE ) . Let subscript “ [ u ] ” mark the quantities corresponding to a small subgraph surrounding node u . For example , the subgraph itself is G [ u ] . For an L-layer GNN , let superscript “ ( ` ) ” denote the layer- ` quantities ( 1 ≤ ` ≤ L ) . Let d ( ` ) be the number of channels for layer ` ; H ( ` −1 ) ∈ R|V|×d ( ` −1 ) andH ( ` ) ∈ R|V|×d ( ` ) be the input and output feature matrices . Thus , H ( 0 ) = X and d ( 0 ) = d. Further , let Y = H ( L ) . The operation of a layer can be abstracted asH ( ` ) = f ( H ( ` −1 ) , A ; W ( ` ) ) , whereW ( ` ) are the learnable weights .

Summary: This paper proposes a new extension of GNNs to deep GNNs, which use subgraphs to keep the computational costs low for training large graphs. It addresses the two main reasons that GNNs have not previously been extended to deep GNNs: expressivity and computational cost. Increasing the number of layers in a GNN leads to averaging over more nodes, which in turn collapses the learned embeddings. The paper claims that using shallow graphs instead of the full graphs avoids this oversmoothing issue. Additionally, using the full graph is computationally expensive since the neighborhood sizes grow with the number of neighbors. Using shallow subgraphs instead allows the size of the neighborhoods to remain constant as the number of layers increase. To this end, the paper presents SHADOW-GNN, a Deep GNN with shallow sampling. They extend this framework to GraphSAGE and GAT models and show that it improves performance over the original model with a lower computational cost. Overall, this method seems well-motivated, and both theoretical and empirical results support their claims. There are a few points on which clarification from the authors would be helpful.