Scalable Graph Neural Networks for Heterogeneous Graphs

1 INTRODUCTION . In recent years , deep learning on graphs has attracted a great deal of interest , with new applications ranging from social networks and recommender systems , to biomedicine , scene understanding , and modeling of physics ( Wu et al. , 2020 ) . One popular branch of graph learning is based on the idea of stacking learned “ graph convolutional ” layers that perform feature transformation and neighbor aggregation ( Kipf & Welling , 2017 ) , and has led to an explosion of variants collectively referred to as Graph Neural Networks ( GNNs ) ( Hamilton et al. , 2017 ; Xu et al. , 2018 ; Velickovic et al. , 2018 ) . Most benchmarks for learning on graphs focus on very small graphs , but the relevance of such models to large-scale social network and e-commerce datasets was quickly recognized ( Ying et al. , 2018 ) . Since the computational cost of training and inference on GNNs scales poorly to large graphs , a number of sampling approaches have been proposed that improve the time and memory cost of GNNs by operating on subsets of graph nodes or edges ( Hamilton et al. , 2017 ; Chen et al. , 2017 ; Zou et al. , 2019 ; Zeng et al. , 2019 ; Chiang et al. , 2019 ) . Recently several papers have argued that on a range of benchmark tasks – social network and e-commerce tasks in particular – GNNs primarily derive their benefits from performing feature smoothing over graph neighborhoods rather than learning non-linear hierarchies of features as implied by the analogy to CNNs ( Wu et al. , 2019 ; NT & Maehara , 2019 ; Chen et al. , 2019 ; Rossi et al. , 2020 ) . Surprisingly , Rossi et al . ( 2020 ) demonstrate that a one-layer MLP operating on concatenated N-hop averaged features , which they call Scalable Inception Graph Network ( SIGN ) , performs competitively with state-of-the-art GNNs on large web datasets while being more scalable and simpler to use than sampling approaches . Neighbor-averaged features can be precomputed , reducing GNN training and inference to a standard classification task . However , in practice the large graphs used in web-scale classification problems are often heterogeneous , encoding many types of relationship between different entities ( Lerer et al. , 2019 ) . While GNNs extend naturally to these multi-relation graphs ( Schlichtkrull et al. , 2018 ) and specialized methods further improve the state-of-the-art on them ( Hu et al. , 2020b ; Wang et al. , 2019b ) , it is not clear how to extend neighbor-averaging approaches like SIGN to these graphs . In this work , we investigate whether neighbor-averaging approaches can be applied to heterogeneous graphs ( HGs ) . We propose Neighbor Averaging over Relation Subgraphs ( NARS ) , which computes neighbor averaged features for random subsets of relation types , and combines them into a single set of features for a classifier using a 1D convolution . We find that this scalable approach exceeds the accuracy of state-of-the-art GNN methods for heterogeneous graphs on tasks in three benchmark datasets . One downside of NARS is that it requires a large amount of memory to store node features for many random subgraphs . We describe an approximate version that fixes the memory scaling issue , and show that it does not degrade accuracy on benchmark tasks . 2 BACKGROUND . Graph Neural Networks are a type of neural model for graph data that uses graph structure to transform input node features into a more predictive representation for a supervised task . A popular flavor of graph neural network consists of stacked layers of operators composed of learned transformations and neighbor aggregation . These “ message-passing ” GNNs were inspired by spectral notions of graph convolution ( Bruna et al. , 2014 ; Defferrard et al. , 2016 ; Kipf & Welling , 2017 ) . Consider a graph G with n vertices and adjacency matrix A ∈ Rn×n . A graph convolution g ? x of node features x by a filter g is defined as a multiplication by g in the graph Fourier basis , just as a standard convolution is a multiplication in Fourier space . The Fourier basis for a graph is defined as the eigenvectors U of the normalized Laplacian , and can be thought of as a basis of functions of varying smoothness over the graph . g ? x = UgUTx ( 1 ) Any convolution g can be approximated by a series of k-th order polynomials in the Laplacian , which depend on neighbors within a k-hop radius ( Hammond et al. , 2011 ) . By limiting this approximation to k = 1 , Kipf & Welling ( 2017 ) arrive at an operation that consists of multiplying node features by the normalized adjacency matrix , i.e . averaging each node ’ s neighbor features . Such an operation can be viewed as a graph convolution by a particular smoothing kernel . A Graph Convolutional Network ( GCN ) is constructed by stacking multiple layers , each with a neighbor averaging step followed by a linear transformation . Many variants of this approach of stacked message-passing layers have since been proposed with different aggregation functions and for different applications ( Velickovic et al. , 2018 ; Xu et al. , 2018 ; Hamilton et al. , 2017 ; Schlichtkrull et al. , 2018 ) . Early GNN work focused on tasks with small graphs ( thousands of nodes ) , and it ’ s not straightforward to scale these methods to large-scale graphs . Applying neighbor aggregation by directly multiplying node features by the sparse adjacency matrix at each training step is computationally expensive and does not permit minibatch training . On the other hand , applying a GCN for a minibatch of labeled nodes requires aggregation over a receptive field ( neighborhood ) of diameter d equal to the GCN depth , which can grow exponentially in d. Recent work in scaling GNNs to very large graphs have focused on training the GNN on sampled subsets of neighbors or subgraphs to allevate the computation and memory cost ( Hamilton et al. , 2017 ; Chen et al. , 2017 ; Zou et al. , 2019 ; Zeng et al. , 2019 ; Chiang et al. , 2019 ) . Rossi et al . ( 2020 ) proposed a different approach to scaling GCNs , called SIGN : As is shown in Figure 2 , by eliding all learned parameters from intermediate layers , the GNN graph aggregation steps can be pre-computed as iterated neighbor feature averages , and model training consists of training an MLP on these neighbor-averaged features . On benchmark tasks on large graphs , they observed that SIGN achieved similar accuracy to state-of-the-art GNNs . The success of SIGN suggests that GNNs are primarily using the graph to “ smooth ” node features over local neighborhoods rather than learning non-linear feature hierarchies . Similar hypotheses have been argued in several other recent works ( Wu et al. , 2019 ; NT & Maehara , 2019 ; Chen et al. , 2019 ) 1 Standard GCNs extend naturally to heterogeneous ( aka relational ) graphs by applying relation-specific learned transformations ( Schlichtkrull et al. , 2018 ) . There have also been a number of GNN variants specialized to heterogeneous graphs . HetGNN ( Zhang et al. , 2019a ) performs fixed-size random walk on the graph and encodes heterogeneous content with RNNs . Heterogeneous Attention Network ( Wang et al. , 2019b ) generalizes neighborhood of nodes based on semantic patterns ( called metapaths ) and extends GAT ( Velickovic et al. , 2018 ) with a semantic attention mechanism . The Heterogeneous Graph Transformer ( Hu et al. , 2020b ) uses an attention mechanism that conditions on node and edge types , and introduces relative temporal encoding to handle dy- namic graphs . These models inherit the scaling limitation of GNN and are expensive to train on large graphs . Therefore , it is of practical importance to generalize the computationally much simpler SIGN model to heterogeneous graphs . 3 NEIGHBOR AVERAGING OVER RELATION SUBGRAPHS FOR HETEROGENEOUS GRAPHS . The challenge with adapting SIGN to heterogeneous graphs is how to incorporate the information about different node and edge types . Relational GNNs like R-GCN naturally handle heterogeneous graphs by learning different transformations for each relation type ( Schlichtkrull et al. , 2018 ) , but SIGN elides these learned transformations . While one can naively apply SIGN to heterogeneous graphs by ignoring entity and relation types , doing so results in poor task performance ( Table 4 ) . In this section , we propose Neighbor Averaging over Relation Subgraphs ( NARS ) . The key idea of NARS is to utilize entity and relation information by repeatedly sampling subsets of relation types and building subgraphs consisting only of edges of these types , which we call relation subgraphs . We then perform neighbor averaging on these relation subgraphs and aggregate features with a learned 1-D convolution . The resulting features are then used to train an MLP , as in SIGN . We take inspiration from the notion of “ metapaths ” proposed in Dong et al . ( 2017 ) and used by several recent heterogeneous GNN approaches ( Wang et al. , 2019b ) . A metapath is a sequence of relation types that describes a semantic relationship among different types of nodes ; for example , one metapath in an academic graph might be “ venue - paper - author ” , which could represent “ venues of papers with the same author as the target paper node ” . Information passed and aggregated along a metapath is expected to be semantically meaningful . In previous work like HAN ( Wang et al. , 2019b ) , features from different metapaths are later aggregated to capture different neighborhood information for each node . In prior work , relevant metapaths were manually specified as input ( Dong et al. , 2017 ; Wang et al. , 2019b ) , but we hypothesized that the same information could be captured by randomly sampling metapaths . However , sampling individual metapaths doesn ’ t scale well to graphs with many edge 1SIGN primarily differs from these other proposed methods by concatenating neighbor-aggregated features from different numbers of hops of feature aggregation . This addresses the need to balance the benefits of feature smoothing from large neighborhoods with the risk of “ oversmoothing ” the features and losing local neighborhood information when GNNs are too deep ( Li et al. , 2018 ) , allowing the classifier to learn a balance between features from different GNN depths . types : for a graph with M edge types , it would require O ( M ) metapaths to even sample each edge type in a single metapath . As a result , one might need to sample a large set of metapaths to obtain good prediction results , and in practice we obtained poor task performance by sampling metapaths . We observe that metapaths are an instance of a more general class of aggregation procedures : those that aggregate at each GNN layer over a subset of relation types rather than a single relation type . We consider a different procedure in that class : sampling a subset of relation types ( uniformly from the power set of relation types ) , and using this subset for all aggregation layers . This procedure amounts to randomly selecting K relation subgraphs each specified by a subset of relations , and performing L-hop neighbor aggregation across each of these subgraphs . We found that this strategy led to strong task performance ( §5 ) , but it ’ s possible that other aggregation strategies could perform even better . Given a heterogeneous graph G and its edge relation type setR , our proposed method first samples K unique subsets from R. Then for each sampled subset Ri ⊆ R , we generate a relation subgraph Gi from G in which only edges whose type belongs to Ri are kept . We treat Gi as a homogeneous graph , and perform neighbor aggregation to generate L-hop neighbor features for each node . Let Hv,0 be the input features ( of dimension D ) for node v. For each subgraph Gi , the l-th hop features Hiv , l are computed as Hiv , l = ∑ u∈Ni ( v ) 1 |Ni ( v ) | Hiu , l−1 ( 2 ) where Ni ( v ) is the set of neighbors of node v in Gi .

Summary: The authors propose a method to broaden the scope of SIGN, a technique recently introduced for single-relational graphs. The method allows SIGN to also be applied to multi-relational graphs (often called heterogeneous or knowledge graphs in different communities). In SIGN, various powers of the Laplacian are precomputed. For each power, the features of the nodes of a node’s neighborhood are averaged and (e.g., with an MLP) projected into a node vector representation. This is then used to classify the node.