NAFS: A Simple yet Tough-to-Beat Baseline for Graph Representation Learning

1 INTRODUCTION . In recent years , graph representation learning has been extensively applied in various application scenarios , such as node clustering , link prediction , node classification , and graph classification ( Kipf & Welling , 2016b ; a ; Hamilton et al. , 2017 ; Bo et al. , 2020 ; Hettige et al. , 2020 ; Wang et al. , 2016 ; Wu et al. , 2020 ; Abu-El-Haija et al. , 2019 ) . The goal of graph representation learning is to encode graph information to node embeddings . Traditional graph representation learning methods , such as DeepWalk ( Perozzi et al. , 2014 ) , Node2vec ( Grover & Leskovec , 2016 ) , LINE ( Tang et al. , 2015 ) , and ComE ( Cavallari et al. , 2017 ) merely focus on preserving graph structure information . GNN-based graph representation learning has attracted intensive interest by combining knowledge from both graph structure and node features . While most of these GNN-based methods are designed based on Graph AutoEncoder ( GAE ) and Variational Graph AutoEncoder ( VGAE ) ( Kipf & Welling , 2016b ) , these methods share two major limitations : Shallow Architecture . Previous work shows that although stacking multiple GNN layers in Graph Convolutional Network ( GCN ) ( Kipf & Welling , 2016a ) is capable of exploiting deep structural information , applying a large number of GNN layers might lead to indistinguishable node embeddings , i.e. , the over-smoothing issue ( Li et al. , 2018 ) . Therefore , most state-of-the-art GNNs resort to shallow architectures , which hinders the model from capturing long-range dependencies . Low Scalability . GNN-based graph representation learning methods can not scale well to large graphs due to the expensive computation cost and high memory usage . Most existing GNNs need to repeatedly perform the computationally expensive and recursive feature smoothing , which involves the participation of the entire graph at each training epoch . Furthermore , most methods adopt the same training loss function as GAE , which introduces high memory usage by storing the dense-form adjacency matrix on GPU . For a graph of size 200 million , its dense-form adjacency matrix requires a space of roughly 150GB , exceeding the memory capacity of the current powerful GPU devices . To tackle these issues , we propose a new graph representation learning method , which is embarrassingly simple : just smooth the node features and then combine the smoothed features in a node-adaptive manner . We name this method node-adaptive feature smoothing ( NAFS ) , and its goal is to construct better node embeddings that integrate the information from both graph structural information and node features . Based on the observation that different nodes have highly diverse “ smoothing speed ” , NAFS adaptively smooths each node feature and takes advantage of both loworder and high-order neighborhood information of each node . In addition , feature ensemble is also employed to combine the smoothed features extracted via different smoothing operators . Since NAFS is training-free , it significantly reduces the training cost and scales better to large graphs than most GNN-based graph representation learning methods . This paper is not meant to diminish the current advancements in GNN-based graph representation learning approaches . Instead , we aim to introduce an easier way to obtain high-quality node embeddings and understand the source of performance gains of these approaches better . Feature smoothing could be a promising direction towards a more simple and effective integration of information from both graph structure and node features . Our contributions are as follows : ( 1 ) New perspective . To the best of our knowledge , we are the first to explore the possibility that simple feature smoothing without any trainable parameters could even outperform state-of-the-art GNNs ; this incredible finding opens up a new direction towards efficient and scalable graph representation learning . ( 2 ) Novel method . We propose NAFS , a nodeadaptive feature smoothing approach along with various feature ensemble strategies , to fully exploit knowledge from both the graph structure and node features . ( 3 ) State-of-the-art performance . We evaluate the effectiveness and efficiency of NAFS on real-world datasets across various graph-based tasks , including node clustering and link prediction . Empirical results demonstrate that NAFS performs comparably with or even outperforms the state-of-the-art GNNs , and achieves up to two orders of magnitude speedup . In particular , on PubMed , NAFS outperforms GAE ( Kipf & Welling , 2016b ) and AGE ( Cui et al. , 2020 ) by a margin of 9.0 % and 3.8 % in terms of NMI in node clustering , while achieving up to 65.4× and 88.6× training speedups , respectively . 2 PRELIMINARY . In this section , we first explain the notations and problem formulation . Then , we review current GNNs and GNN-based graph representation learning . 2.1 NOTATIONS AND PROBLEM FORMULATION .. In this paper , we consider an undirected graph G = ( V , E ) with |V| = n nodes and |E| = m edges . Here we suppose that m ∝ n as it is the case in most real-world graphs . We denote by A the adjacency matrix of G. Each node can possibly have a feature vector of size f , which stacks up to an n× f feature matrix X . The degree matrix of A is denoted as D = diag ( d1 , d2 , · · · , dn ) ∈ Rn×n , where di = ∑ vj∈V Aij . We denote the final node embedding matrix as Z , and evaluate it in both the node clustering and the link prediction tasks . The node clustering task requires the model to partition the nodes into c disjoint groups G1 , G2 , · · · , Gc , where similar nodes should be in the same group . The target of the link prediction task is to predict whether an edge exists between given node pairs . 2.2 GRAPH CONVOLUTIONAL NETWORK .. Based on the assumption that locally connected nodes are likely to enjoy high similarity ( McPherson et al. , 2001 ) , each node in most GNN models iteratively smooths the representations of its neighbors for better node embedding . Below is the formula of the l-th GCN layer ( Kipf & Welling , 2016a ) : X ( l ) = δ ( ÂX ( l−1 ) Θ ( l ) ) , Â = D̃−1/2ÃD̃−1/2 , Ã = A + In , ( 1 ) where X ( l ) is the node embedding matrix at layer l , X ( 0 ) is the original feature matrix , Θ ( l ) are the trainable weights , and δ is the activation function . Â is the smoothing matrix that helps each node to smooth representations of neighboring nodes . As shown in Eq . 1 , each GCN layer contains two operations : feature aggregation ( smoothing ) and feature transformation . Figure 1 shows the framework of a two-layer GCN . The l-th layer in GCN firstly executes feature smoothing on the node embedding X ( l−1 ) . Then , the smoothed feature X̃ ( l−1 ) is transformed with trainable weights Θ ( l ) and activation function δ to generate new node embedding X ( l ) . Note that GCN will degrade to MLP if feature smoothing is removed from each layer . 2.3 GNN-BASED GRAPH REPRESENTATION LEARNING .. GAE ( Kipf & Welling , 2016b ) , the first and the most representative GNN-based graph embedding method , adopts an encoder to generate node embedding matrix Z with inputs Â and X . A simple inner product decoder is then used to reconstruct the adjacency matrix . The final training loss of GAE is the binary entropy loss between A′ and Ã , the reconstructed adjacency matrix and the original adjacency matrix with self loop added . A′ = sigmoid ( Z · ZT ) , L = ∑ 1≤i , j≤n −Ãi , j logA′i , j − ( 1− Ãi , j ) log ( 1−A′i , j ) ) . ( 2 ) Motivated by GAE , lots of GNN-based graph representation learning methods are proposed recently . MGAE ( Wang et al. , 2017 ) presents a denoising marginalized autoencoder that reconstructs the node feature matrix X. ARGA ( Pan et al. , 2018 ) adopts the adversarial learning strategy , and its generated node embeddings are forced to match a prior distribution . DAEGC ( Wang et al. , 2019 ) exploits side information to generate node embeddings in a self-supervised way . AGC ( Zhang et al. , 2019 ) proposes an improved filter matrix to better filter out the high-frequency noise . AGE ( Cui et al. , 2020 ) further improves AGC by using the similarity of embedding rather than the adjacency matrix to consider original node feature information . Compared with GNN-based graph representation learning methods that rely on the trainable parameters to learn node embeddings , our NAFS is training-free and thus enjoys higher efficiency and scalability . 3 OBSERVATION AND INSIGHT . In this section , we make a quantitative analysis on the over-smoothing issue at the node level and then provide some insights when designing NAFS on graphs . 3.1 FEATURE SMOOTHING IN DECOUPLED GNNS . Recently , many works ( Wu et al. , 2019 ; Zhu & Koniusz , 2021 ; Chen et al. , 2020 ; Zhang et al. , 2021 ) propose to decouple the feature smoothing and feature transformation in each GCN layer for scalable node classification . Concretely , they execute the feature smoothing operation in advance , and the smoothed features are then fed into a simple MLP to generate the final predicted node labels . Under this framework , the predictive node classification accuracy of these methods is comparable with or even higher than the one of coupled GNNs , and these works claim that the true power of GNNs lies in feature smoothing rather than feature transformation . We split the framework of these decoupled GNNs into two parts : feature smoothing and MLP training . Feature smoothing aims to combine the graph structural information and node features into better features for the subsequent MLP ; while MLP training only takes in the smoothed feature and is specially trained for a given task . As stated by previous decoupled GNNs ( Wu et al. , 2019 ; Zhu & Koniusz , 2021 ) , the true success of GNNs lies in feature smoothing rather than feature transformation . Correspondingly , we propose to remove feature transformation and preserve the key feature smoothing part alone for simple and scalable node representation . There is another branch of GNNs that also decouple the feature smoothing and feature transformation . The most representative method of this category is APPNP ( Klicpera et al. , 2018 ) . It first feeds the raw node features into an MLP to generate intermediate node embeddings ; then the personalized PageRank based propagation operations are performed on the node embeddings to produce final prediction results . However , compared with scalable decoupled GNNs mentioned in the previous paragraph , this branch of GNNs still have to recursively execute propagation operations in each training epoch , which makes it impossible to perform on large-scale graphs . In the remaining part of this paper , the terminology “ decoupled GNNs ” refers particularly to the scalable decoupled GNNs mentioned in the previous two paragraphs . 3.2 MEASURING SMOOTHING LEVEL . To capture deep graph structural information , a straightforward way is to simply stack multiple GNN layers . However , a large number of feature smoothing operations in a GNN model would lead to indistinguishable node embeddings , i.e. , the over-smoothing issue ( Li et al. , 2018 ) . Concretely , if we execute ÂX for infinite times , the node embeddings within the same connected component would reach a stationary state . When adopting Â = D̃r−1ÃD̃−r , Â∞ follows Â∞i , j = ( di + 1 ) r ( dj + 1 ) 1−r 2m+ n , ( 3 ) which shows that the influence from node vi to vj is only determined by their degrees . Under the extreme condition that r = 0 , all the nodes within one connected component have exactly the same representation , making it impossible to apply the node embeddings to subsequent tasks . Here we introduce a new metric , “ Over-smoothing Distance ” , to measure each node ’ s smoothing level . A smaller value indicates that the node is closer to the stationary state , i.e. , closer to over-smoothing . Definition 3.1 ( Over-smoothing Distance ) . The Over-smoothing Distance Di ( k ) parameterized by node i and smoothing step k is defined as Di ( k ) = Dis ( [ Â kX ] i , [ Â ∞X ] i ) , ( 4 ) where [ ÂkX ] i denotes the ith row of ÂkX , representing the representations of node vi after smoothing k times ; [ Â∞X ] i denotes the ith row of Â∞X , representing the stationary state of node vi ; Dis ( · ) is a distance function or a function positively relative with the difference , which can be implemented using Euclidean distance , the inverse of cosine similarity , etc .

Summary: The authors of this paper took a novel perspective to present the node-adaptive feature smoothing (NAFS) algorithm, which generates node embeddings without explicit training/parameter learning. The method first performs feature smoothing, then combines the smoothed features using adaptive weights which are node-specific. They further enhanced this method by ensembling the smoothed features extracted with different hyper-parameters. The authors have conducted many experiments to validate the model performance, and demonstrate the model's efficiency empirically.