Graph Attention Multi-layer Perceptron

1 INTRODUCTION . Graph Neural Networks ( GNNs ) generalize convolutional neural networks to graph-structured data and have achieved great success in a wide range of tasks , including node classification , link prediction , and recommendation . ( Kipf & Welling , 2016 ; Hamilton et al. , 2017 ; Bo et al. , 2020 ; Cui et al. , 2020 ; Fan et al. , 2019 ) . Through stacking K graph convolution layers , GNNs learn node representations by utilizing information from the K-hop neighborhood and thus enhance the performance by getting more unlabeled nodes involved in the training process . In such a GNN model , the nodes within the K-hop neighborhood of a specific node are called this node ’ s Receptive Field ( RF ) . As the size of RF grows exponentially to the number of GNN layers , the rapidly expanding RF incurs high computation and memory costs in a single machine . Besides , even in a distributed environment , GNN has to pull a great number of neighboring node features to compute the representation of each node , leading to high communication cost ( Zheng et al. , 2020 ) . Many recent advancements towards scalable GNNs are based on model simplification . For example , Simplified GCN ( SGC ) ( Wu et al. , 2019 ) decouples the feature propagation and the non-linear transformation process , and the former is executed during pre-processing . Unlike the sampling-based methods ( Hamilton et al. , 2017 ) , which still need feature propagation during each training epoch , this time-consuming process in SGC is only executed once , and only the nodes of the training set are involved in the training process . As a result , SGC is computation and memory-efficient in a single machine and scalable in distributed settings since it does not require each machine to fetch neighboring node features during the model training process . Despite the high efficiency and scalability , SGC simply preserves a fixed RF for all the nodes by assigning them the same feature propagation depth . Such a fixed propagation mechanism in SGC disables its ability to exploit knowledge within neighborhoods of different sizes . Lines of other simplified models have been proposed to learn better node representations exploiting multi-scale knowledge . SIGN ( Frasca et al. , 2020 ) proposes to concatenate all the propagated features without information loss , while S2GC ( Zhu & Koniusz , 2021 ) averages all these propagated features to generate the combined feature . Although multi-scale knowledge is considered , the importance and correlations between multiple scales are ignored . Being the first attempt to explore the correlations between different scales of knowledge , GBP ( Chen et al. , 2020b ) adopts a heuristic constant decay factor for the weighted average for propagated features at different propagation steps . Motivated by Personalized PageRank , the large-scale features has a higher risk of over-smoothing , and they will contribute less to the combination in GBP . Unfortunately , the coarse-grained , layer-wise combination prevents these methods from unleashing their full potential . As shown in Figure 1 ( a ) , different nodes require different propagation steps to achieve optimal predictive accuracy . Besides , assigning the same weight distribution to propagated features along with propagation depth to all the nodes may be unsuitable due to the inconsistent RF expansion speed shown in Figure 1 ( b ) . However , nodes in most existing GNNs are restricted to a fixed-hop neighborhood and insensitive to the actual demands of different nodes . This imperfection either makes that long-range dependencies can not be fully leveraged due to limited hops/layers or loses local information by introducing many irrelevant nodes into the receptive fields for many nodes when increasing the number of propagation depth ( Chen et al. , 2020a ; Li et al. , 2018 ; Xu et al. , 2018 ) . The above observations motivate us to explicitly learn the importance and correlation of multiscale knowledge in a node-adaptive manner . To this end , we develop a new architecture – Graph Attention Multi-Layer Perceptron ( GAMLP ) – that could automatically exploit the knowledge over different neighborhoods at the granularity of nodes . GAMLP achieves this by introducing two novel attention mechanisms : Recursive attention and Jumping Knowledge ( JK ) attention . These two attention mechanisms can capture the complex correlations between propagated features at different propagation depths in a node-adaptive manner . Consequently , our architecture has the same benefits as the existing simplified and scalable GNN models while providing much better performance derived from its ability to utilizes a node-adaptive receptive field . Moreover , the proposed attention mechanisms can be applied to both node features and labels over neighborhoods with different sizes . By combining these two categories of information together , GAMLP could achieve the best of both worlds in terms of accuracy . Our contributions are as follows : ( 1 ) New perspective . To the best of our knowledge , we are the first to explore both node-adaptive feature and label propagation schemes for scalable GNNs . ( 2 ) Novel method . We propose GAMLP , a scalable , efficient , and deep graph model . ( 3 ) State-of-the-art performance . Experimental results demonstrate that GAMLP achieves state-of-theart performance on 12 benchmark datasets while maintains high scalability and efficiency . In particular , GAMLP outperforms the competitive baseline GraphSAINT ( Zeng et al. , 2020 ) in terms of accuracy by a margin of 0.42 % , 3.02 % and 0.44 % on PPI , Flickr , and Reddit datasets under the inductive setting , while achieving up to 45× training speedups in the large ogbn-products dataset . Remarkably , under the transductive setting in large OGB datasets , the accuracy of GAMLP exceeds the current state-of-the-art method by 1.03 % and 1.32 % on the ogbn-products and ogbn-papers100M datasets , respectively . 2 PRELIMINARIES . 2.1 PROBLEM FORMULATION . We consider an undirected graph G = ( V , E ) with |V| = n nodes , |E| = m edges , and c different node classes . We denote by A the adjacency matrix of G , weighted or not . Nodes can possibly have features vector of size f , stacked up in an n × f matrix X . D = diag ( d1 , d2 , · · · , dN ) ∈ Rn×n denotes the degree matrix of A , where di = ∑ vj∈V Aij is the degree of node vi . Suppose Vl is the labeled set , and our goal is to predict the labels for nodes in the unlabeled set Vu with the supervision of Vl . 2.2 SCALABLE GNNS . Sampling . A commonly used method to tackle the scalability issue ( i.e. , the recursive neighborhood expansion ) in GNN is sampling . As a node-wise sampling method , GraphSAGE ( Hamilton et al. , 2017 ) randomly samples a fixed-size set of neighbors for computation in each mini-batch . VRGCN ( Chen et al. , 2018a ) analyzes the variance reduction , and it reduces the size of samples with additional memory cost . For the layer-wise sampling , Fast-GCN ( Chen et al. , 2018b ) samples a fixed number of nodes at each layer , and ASGCN ( Huang et al. , 2018 ) proposes the adaptive layer-wise sampling with better variance control . In the graph level , Cluster-GCN ( Chiang et al. , 2019 ) firstly clusters the nodes and then samples the nodes in the clusters , and GraphSAINT ( Zeng et al. , 2020 ) directly samples a subgraph for mini-batch training . Orthogonal to model simplification , sampling has already been widely used in many GNNs and GNN systems ( Zheng et al. , 2020 ; Zhu et al. , 2019 ; Fey & Lenssen , 2019 ) . However , these sampling-based GNNs are imperfect because they still face high communication costs , and the sampling quality highly influences the model performance . Graph-wise Propagation . Recently studies have observed that non-linear feature transformation contributes little to the performance of the GNNs as compared to feature propagation . Thus , a new direction recently emerging for scalable GNN is based on the simplified GCN ( SGC ) ( Wu et al. , 2019 ) , which successively removes nonlinearities and collapsing weight matrices between consecutive layers . This reduces GNNs into a linear model operating on K-layers propagated features : X ( K ) = ÂKX ( 0 ) , Y = softmax ( ΘX ( K ) ) , ( 1 ) where X ( 0 ) = X , X ( K ) is the K-layers propagated feature , and Â = D̃r−1ÃD̃−r . By setting r = 0.5 , 1 and 0 , Â represents the symmetric normalization adjacency matrix D̃−1/2ÃD̃−1/2 ( Klicpera et al. , 2019 ) , the transition probability matrix ÃD̃−1 ( Zeng et al. , 2020 ) , or the reverse transition probability matrix D̃−1Ã ( Xu et al. , 2018 ) , respectively . As the propagated features X ( K ) can be precomputed , SGC is more scalable and efficient for the large graph . However , such graph-wise propagation restricts the same propagation steps and a fixed RF for each node . Therefore , some nodes ’ features may be over-smoothed or under-smoothed due to the inconsistent RF expansion speed , leading to non-optimal performance . Layer-wise Propagation . Following SGC , some recent methods adopt layer-wise propagation to combine the features with different propagation layers . SIGN ( Frasca et al. , 2020 ) proposes to concatenate the propagated features at different propagation depth after simple linear transformation : [ X ( 0 ) W0 , X ( 1 ) W1 , ... , X ( K ) WK ] . S2GC ( Zhu & Koniusz , 2021 ) proposes the simple spectral graph convolution to average the propagated features in different iterations as X ( K ) = K∑ l=0 ÂlX ( 0 ) . In addition , GBP ( Chen et al. , 2020b ) further improves the combination process by weighted averaging as X ( K ) = K∑ l=0 wlÂ lX ( 0 ) with the layer weight wl = β ( 1− β ) l. Similar to these works , we also use a linear model for higher training scalability . The difference lies in that we consider the propagation process from a node-wise perspective and each node in GAMLP has a personalized combination of different steps of the propagated features . 2.3 LABEL UTILIZATION ON GNNS .. Labels of training nodes are conventionally only used as supervision signals in loss functions in most graph learning methods . However , there also exist some graph learning methods that directly exploit the labels of training nodes . Among them , the label propagation algorithm ( Zhu & Ghahramani , 2002 ) is the most well-known one . It simply regards the partially observed label matrix Y ∈ RN×C as input features for nodes in the graph and propagates the input features through the graph structure , where C is the number of candidate classes . UniMP ( Shi et al. , 2020 ) proposes to map the partially observed label matrix Y to the dimension of the node feature matrix X and add these two matrices together as the new input feature . To fight against the label leakage problem , UniMP further randomly masks the training nodes during every training epoch . Instead of using only the hard training labels , Correct & Smooth ( Huang et al. , 2020 ) first trains a simple model such as an MLP and gets this model ’ s predicted soft labels for unlabeled nodes . Then , it propagates the learning errors on the labeled nodes to connected nodes and smooths the output in a Personalized PageRank manner like APPNP ( Klicpera et al. , 2019 ) . Besides , SLE ( Sun & Wu , 2021 ) decouples the label utilization procedure in UniMP , and executes the propagation in advance . Unlike UniMP , “ label reuse ” ( Wang et al. , 2021 ) concatenates the partially observed label matrix Y with the node feature matrix X to form the new input matrix . Concretely , it fills the missing elements in the partially observed label matrix Y with the soft label predicted by the model , and this newly generated Y′ is again concatenated with X and then fed into the model to generate new predictions .

Summary: The paper proposed the Graph Attention Multi-Layer Perceptron (GAMLP) model that combines features extracted from two streams: 1) node-adaptive features that are obtained via "Graph-wise / layer-wise propagation" (Section 2.2), 2) features obtained from label propagation. The author also proposed two ways for performing the node-adaptive attention: JK attention, and Recursive attention (Section 3.3). Extensive experiments are conducted on 9 transductive datasets and 3 inductive datasets. The results show that the proposed GAMLP(JK) and GAMLP(R) can outperform the baselines.