Node Dependent Local Smoothing for Scalable Graph Learning

Recent works reveal that feature or label smoothing lies at the core of Graph Neural Networks ( GNNs ) . Concretely , they show feature smoothing combined with simple linear regression achieves comparable performance with the carefully designed GNNs , and a simple MLP model with label smoothing of its prediction can outperform the vanilla GCN . Though an interesting finding , smoothing has not been well understood , especially regarding how to control the extent of smoothness . Intuitively , too small or too large smoothing iterations may cause under-smoothing or over-smoothing and can lead to sub-optimal performance . Moreover , the extent of smoothness is node-specific , depending on its degree and local structure . To this end , we propose a novel algorithm called node-dependent local smoothing ( NDLS ) , which aims to control the smoothness of every node by setting a nodespecific smoothing iteration . Specifically , NDLS computes influence scores based on the adjacency matrix and selects the iteration number by setting a threshold on the scores . Once selected , the iteration number can be applied to both feature smoothing and label smoothing . Experimental results demonstrate that NDLS enjoys high accuracy – state-of-the-art performance on node classifications tasks , flexibility – can be incorporated with any models , scalability and efficiency – can support large scale graphs with fast training . 1 Introduction . In recent years , Graph Neural Networks ( GNNs ) have received a surge of interest with the state-of-theart performance on many graph-based tasks [ 2 , 36 , 9 , 34 , 28 , 29 ] . Recent works have found that the success of GNNs can be mainly attributed to smoothing , either at feature or label level . For example , SGC [ 27 ] shows using smoothed features as input to a simple linear regression model achieves comparable performance with lots of carefully designed and complex GNNs . At the smoothing stage , features of neighbor nodes are aggregated and combined with the current node ’ s feature to form smoothed features . This process is often iterated multiple times . The smoothing is based on the assumption that labels of nodes that are close to each other are highly correlated , therefore , the features of nodes nearby should help predict the current node ’ s label . One crucial and interesting parameter of neighborhood feature aggregation is the number of smoothing iterations k , which controls how much information is being gathered . Intuitively , an aggregation process of k iterations ( or layers ) enables a node to leverage information from nodes that are k-hop away [ 22 , 33 ] . The choice of k is closely related to the structural properties of graphs and has a 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . significant impact on the model performance . However , most existing GNNs only consider the fixed-length propagation paradigm – a uniform k for all the nodes . This is problematic since the number of iterations should be node dependent based on its degree and local structures . For example , as shown in Figure 1 ( a ) , the two nodes have rather different local structures , with the left red one resides in the center of a dense cluster and the right red one on the periphery with few connections . The number of iterations to reach an optimal level of smoothness are rather different for the two nodes . Ideally , poorly connected nodes ( e.g. , the red node on the right ) needs large iteration numbers to efficiently gather information from other nodes while well-connected nodes ( e.g. , the red node on the left ) should keep the iteration number small to avoid over-smoothing . Though some learning-based approaches have proposed to adaptively aggregate information for each node through gate/attention mechanism or reinforcement learning [ 25 , 17 , 35 , 23 ] , the performance gains are at the cost of increased training complexity , hence not suitable for scalable graph learning . In this paper , we propose a simple yet effective solution to this problem . Our approach , called node-dependent local smoothing ( NDLS ) , calculates a node-specific iteration number for each node , referred to as local smooth iteration ( LSI ) . Once the LSI for a specific node is computed , the corresponding local smoothing algorithm only aggregates the information from the nodes within a distance less than its LSI as the new feature . The LSI is selected based on influence scores , which measure how other nodes influence the current node . NDLS sets the LSI for a specific node to be the minimum number of iterations so that the influence score is ϵ-away from the over-smoothing score , defined as the influence score at infinite iteration . The insight is that each node ’ s influence score should be at a reasonable level . Since the nodes with different local structures have different “ smoothing speed ” , we expect the iteration number to be adaptive . Figure 1 ( b ) illustrates Cumulative Distribution Function ( CDF ) for the LSI of individual nodes in real-world graphs . The heterogeneous and long-tail property exists in all the datasets , which resembles the characteristics of the degree distribution of nodes in real graphs . Based on NDLS , we propose a new graph learning algorithm with three stages : ( 1 ) feature smoothing with NDLS ( NDLS-F ) ; ( 2 ) model training with smoothed features ; ( 3 ) label smoothing with NDLS ( NDLS-L ) . Note that in our framework , the graph structure information is only used in pre-processing and post-processing steps , i.e. , stages ( 1 ) and ( 3 ) ( See Figure 2 ) . Our NDLS turns a graph learning problem into a vanilla machine learning problem with independent samples . This simplicity enables us to train models on larger-scale graphs . Moreover , our NDLS kernel can act as a drop-in replacement for any other graph kernels and be combined with existing models such as Multilayer Perceptron ( MLP ) , SGC [ 27 ] , SIGN [ 24 ] , S2GC [ 37 ] and GBP [ 5 ] . Extensive evaluations on seven benchmark datasets , including large-scale datasets like ogbnpapers100M [ 13 ] , demonstrates that NDLS achieves not only the state-of-the-art node classification performance but also high training scalability and efficiency . Especially , NDLS outperforms APPNP [ 25 ] and GAT [ 26 ] by a margin of 1.0 % -1.9 % and 0.9 % -2.4 % in terms of test accuracy , while achieving up to 39× and 186× training speedups , respectively . 2 Preliminaries . In this section , we first introduce the semi-supervised node classification task and review the prior models , based on which we derive our method in Section 3 . Consider a graph G = ( V , E ) with |V| = n nodes and |E| = m edges , the adjacency matrix ( including self loops ) is denoted as Ã ∈ Rn×n and the feature matrix is denoted as X = { x1 , x2 ... , xn } in which xi ∈ Rf represents the feature vector of node vi . Besides , Y = { y1 , y2 ... , yl } is the initial label matrix consisting of one-hot label indicator vectors . The goal is to predict the labels for nodes in the unlabeled set Vu with the supervision of labeled set Vl . GCN smooths the representation of each node via aggregating its own representations and the ones of its neighbors ’ . This process can be defined as X ( k+1 ) = δ ( ÂX ( k ) W ( k ) ) , Â = D̃r−1ÃD̃−r , ( 1 ) where Â is the normalized adjacency matrix , r ∈ [ 0 , 1 ] is the convolution coefficient , and D̃ is the diagonal node degree matrix with self loops . Here X ( k ) and X ( k+1 ) are the smoothed node features of layer k and k + 1 respectively while X ( 0 ) is set to X , the original feature matrix . In addition , W ( k ) is a layer-specific trainable weight matrix at layer k , and δ ( · ) is the activation function . By setting r = 0.5 , 1 and 0 , the convolution matrix D̃r−1ÃD̃−r represents the symmetric normalization adjacency matrix D̃−1/2ÃD̃−1/2 [ 16 ] , the transition probability matrix ÃD̃−1 [ 32 ] , and the reverse transition probability matrix D̃−1Ã [ 30 ] , respectively . SGC . For each GCN layer defined in Eq . 1 , if the non-linear activation function δ ( · ) is an identity function and W ( k ) is an identity matrix , we get the smoothed feature after k-iterations propagation as X ( k ) = ÂkX . Recent studies have observed that GNNs primarily derive their benefits from performing feature smoothing over graph neighborhoods rather than learning non-linear hierarchies of features as implied by the analogy to CNNs [ 21 , 8 , 12 ] . By hypothesizing that the non-linear transformations between GCN layers are not critical , SGC [ 27 ] first extracts the smoothed features X ( k ) then feeds them to a linear model , leading to higher scalability and efficiency . Following the design principle of SGC , piles of works have been proposed to further improve the performance of SGC while maintaining high scalability and efficiency , such as SIGN [ 24 ] , S2GC [ 37 ] and GBP [ 5 ] . Over-Smoothing [ 18 ] issue . By continually smoothing the node feature with infinite number of propagation in SGC , the final smoothed feature X ( ∞ ) is X ( ∞ ) = Â∞X , Â∞i , j = ( di + 1 ) r ( dj + 1 ) 1−r 2m+ n , ( 2 ) where Â∞ is the final smoothed adjacency matrix , Â∞i , j is the weight between nodes vi and vj , di and dj are the node degrees for vi and vj , respectively . Eq . ( 2 ) shows that as we smooth the node feature with an infinite number of propagations in SGC , the final feature is over-smoothed and unable to capture the full graph structure information since it only relates with the node degrees of target nodes and source nodes . For example , if we set r = 0 or 1 , all nodes will have the same smoothed features because only the degrees of the source or target nodes have been considered . 3 Local Smoothing Iteration ( LSI ) . The features after k iterations of smoothing is X ( k ) = ÂkX . Inspired by [ 30 ] , we measure the influence of node vj on node vi by measuring how much a change in the input feature of vj affects the representation of vi after k iterations . For any node vi , the influence vector captures the influences of all other nodes . Considering the hth feature of X , we define an influence matrix Ih ( k ) : Ih ( k ) ij = ∂X̂ ( k ) ih ∂X̂ ( 0 ) jh . ( 3 ) I ( k ) = Âk , Ĩi = Â ∞ ( 4 ) Since Ih ( k ) is independent to h , we replace Ih ( k ) with I ( k ) , which can be further represented as I ( k ) = Ih ( k ) , ∀h ∈ { 1 , 2 , .. , f } , where f indicates the number of features of X . We denote I ( k ) i as the ith row of I ( k ) , and Ĩ as I ( ∞ ) . Given the normalized adjacency matrix Â , we can have I ( k ) = Âk and Ĩ = Â∞ . According to Eq . ( 2 ) , Ĩ converges to a unique stationary matrix independent of the distance between nodes , resulting in that the aggregated features of nodes are merely relative with their degrees ( i.e. , over-smoothing ) . We denote I ( k ) i as the ith row of I ( k ) , and it means the influence from the other nodes to the node vi after k iterations of propagation . We introduce a new concept local smoothing iteration ( parameterized by ϵ ) , which measures the minimal number of iterations k required for the influence of other nodes on node vi to be within an ϵ-distance to the over-smoothing stationarity Ĩi . Definition 3.1 . Local-Smoothing Iteration ( LSI , parameterized by ϵ ) is defined as K ( i , ϵ ) = min { k : ||Ĩi − I ( k ) i||2 < ϵ } , ( 5 ) where || · ||2 is two-norm , and ϵ is an arbitrary small constant with ϵ > 0 . Here ϵ is a graph-specific parameter , and a smaller ϵ indicates a stronger smoothing effect . The ϵ-distance to the over-smoothing stationarity Ĩi ensures that the smooth effect on node vi is sufficient and bounded to avoid over-smoothing . As shown in Figure 1 ( b ) , we can have that the distribution of LSI owns the heterogeneous and long-tail property , where a large percentage of nodes have much smaller LSI than the rest . Therefore , the required LSI to approach the stationarity is heterogeneous across nodes . Now we discuss the connection between LSI and node local structure , showcasing nodes in the sparse region ( e.g. , both the degrees of itself and its neighborhood are low ) can greatly prolong the iteration to approach over-smoothing stationarity . This heterogeneity property is not fully utilized in the design of current GNNs , leaving the model design in a dilemma between unnecessary iterations for a majority of nodes and insufficient iterations for the rest of nodes . Hence , by adaptively choosing the iteration based on LSI for different nodes , we can significantly improve model performance . Theoretical Properties of LSI . We now analyze the factors determining the LSI of a specific node . To facilitate the analysis , we set the coefficient r = 0 for the normalized adjacency matrix Â in Eq . ( 1 ) , thus Â = D̃−1Ã . The proofs of following theorems can be found in Appendix A.1 . Theorem 3.1 . Given feature smoothing X ( k ) = ÂkX with Â = D̃−1Ã , we have K ( i , ϵ ) ≤ logλ2 ϵ √ d̃i 2m+ n , ( 6 ) where λ2 is the second largest eigenvalue of Â , d̃i denotes the degree of node vi plus 1 ( i.e. , d̃i = di + 1 ) , and m , n denote the number of edges and nodes respectively . Note that λ2 ≤ 1 . Theorem 3.1 shows that the upper-bound of the LSI is positively correlated with the scale of the graph ( m , n ) , the sparsity of the graph ( small λ2 means strong connection and low sparsity , and vice versa ) , and negatively correlated with the degree of node vi . Theorem 3.2 . For any nodes i in a graph G , K ( i , ϵ ) ≤ max { K ( j , ϵ ) , j ∈ N ( i ) } + 1 , ( 7 ) where N ( i ) is the set of node vi ’ s neighbours . Theorem 3.2 indicates that the difference between two neighboring nodes ’ LSIs is no more than 1 , therefore the nodes with a super-node as neighbors ( or neighbor ’ s neighbors ) may have small LSIs . That is to say , the sparsity of the local area , where a node locates , also affects its LSI positively . Considering Theorems 3.1 and 3.2 together , we can have a union upper-bound of K ( i , ϵ ) as K ( i , ϵ ) ≤ min max { K ( j , ϵ ) , j ∈ N ( i ) } + 1 , logλ2 ϵ √ d̃i 2m+ n . ( 8 )

Summary: This work proposes NDLS, a strategy to estimate a different number of 'propagations' for each node in the graph, delimiting the neighborhoods used to update node embeddings. The authors apply NDLS both as a pre-processing (feature space) and post-processing (prediction space) technique. Between pre-processing and post-processing, any vanilla (non-Graph) classifier can be used.