NAFS: A Simple yet Tough-to-Beat Baseline for Graph Representation Learning

1 INTRODUCTION . In recent years , graph representation learning has been extensively applied in various application scenarios , such as node clustering , link prediction , node classification , and graph classification ( Kipf & Welling , 2016b ; a ; Hamilton et al. , 2017 ; Bo et al. , 2020 ; Hettige et al. , 2020 ; Wang et al. , 2016 ; Wu et al. , 2020 ; Abu-El-Haija et al. , 2019 ) . The goal of graph representation learning is to encode graph information to node embeddings . Traditional graph representation learning methods , such as DeepWalk ( Perozzi et al. , 2014 ) , Node2vec ( Grover & Leskovec , 2016 ) , LINE ( Tang et al. , 2015 ) , and ComE ( Cavallari et al. , 2017 ) merely focus on preserving graph structure information . GNN-based graph representation learning has attracted intensive interest by combining knowledge from both graph structure and node features . While most of these GNN-based methods are designed based on Graph AutoEncoder ( GAE ) and Variational Graph AutoEncoder ( VGAE ) ( Kipf & Welling , 2016b ) , these methods share two major limitations : Shallow Architecture . Previous work shows that although stacking multiple GNN layers in Graph Convolutional Network ( GCN ) ( Kipf & Welling , 2016a ) is capable of exploiting deep structural information , applying a large number of GNN layers might lead to indistinguishable node embeddings , i.e. , the over-smoothing issue ( Li et al. , 2018 ) . Therefore , most state-of-the-art GNNs resort to shallow architectures , which hinders the model from capturing long-range dependencies . Low Scalability . GNN-based graph representation learning methods can not scale well to large graphs due to the expensive computation cost and high memory usage . Most existing GNNs need to repeatedly perform the computationally expensive and recursive feature smoothing , which involves the participation of the entire graph at each training epoch . Furthermore , most methods adopt the same training loss function as GAE , which introduces high memory usage by storing the dense-form adjacency matrix on GPU . For a graph of size 200 million , its dense-form adjacency matrix requires a space of roughly 150GB , exceeding the memory capacity of the current powerful GPU devices . To tackle these issues , we propose a new graph representation learning method , which is embarrassingly simple : just smooth the node features and then combine the smoothed features in a node-adaptive manner . We name this method node-adaptive feature smoothing ( NAFS ) , and its goal is to construct better node embeddings that integrate the information from both graph structural information and node features . Based on the observation that different nodes have highly diverse “ smoothing speed ” , NAFS adaptively smooths each node feature and takes advantage of both loworder and high-order neighborhood information of each node . In addition , feature ensemble is also employed to combine the smoothed features extracted via different smoothing operators . Since NAFS is training-free , it significantly reduces the training cost and scales better to large graphs than most GNN-based graph representation learning methods . This paper is not meant to diminish the current advancements in GNN-based graph representation learning approaches . Instead , we aim to introduce an easier way to obtain high-quality node embeddings and understand the source of performance gains of these approaches better . Feature smoothing could be a promising direction towards a more simple and effective integration of information from both graph structure and node features . Our contributions are as follows : ( 1 ) New perspective . To the best of our knowledge , we are the first to explore the possibility that simple feature smoothing without any trainable parameters could even outperform state-of-the-art GNNs ; this incredible finding opens up a new direction towards efficient and scalable graph representation learning . ( 2 ) Novel method . We propose NAFS , a nodeadaptive feature smoothing approach along with various feature ensemble strategies , to fully exploit knowledge from both the graph structure and node features . ( 3 ) State-of-the-art performance . We evaluate the effectiveness and efficiency of NAFS on real-world datasets across various graph-based tasks , including node clustering and link prediction . Empirical results demonstrate that NAFS performs comparably with or even outperforms the state-of-the-art GNNs , and achieves up to two orders of magnitude speedup . In particular , on PubMed , NAFS outperforms GAE ( Kipf & Welling , 2016b ) and AGE ( Cui et al. , 2020 ) by a margin of 9.0 % and 3.8 % in terms of NMI in node clustering , while achieving up to 65.4× and 88.6× training speedups , respectively . 2 PRELIMINARY . In this section , we first explain the notations and problem formulation . Then , we review current GNNs and GNN-based graph representation learning . 2.1 NOTATIONS AND PROBLEM FORMULATION .. In this paper , we consider an undirected graph G = ( V , E ) with |V| = n nodes and |E| = m edges . Here we suppose that m ∝ n as it is the case in most real-world graphs . We denote by A the adjacency matrix of G. Each node can possibly have a feature vector of size f , which stacks up to an n× f feature matrix X . The degree matrix of A is denoted as D = diag ( d1 , d2 , · · · , dn ) ∈ Rn×n , where di = ∑ vj∈V Aij . We denote the final node embedding matrix as Z , and evaluate it in both the node clustering and the link prediction tasks . The node clustering task requires the model to partition the nodes into c disjoint groups G1 , G2 , · · · , Gc , where similar nodes should be in the same group . The target of the link prediction task is to predict whether an edge exists between given node pairs . 2.2 GRAPH CONVOLUTIONAL NETWORK .. Based on the assumption that locally connected nodes are likely to enjoy high similarity ( McPherson et al. , 2001 ) , each node in most GNN models iteratively smooths the representations of its neighbors for better node embedding . Below is the formula of the l-th GCN layer ( Kipf & Welling , 2016a ) : X ( l ) = δ ( ÂX ( l−1 ) Θ ( l ) ) , Â = D̃−1/2ÃD̃−1/2 , Ã = A + In , ( 1 ) where X ( l ) is the node embedding matrix at layer l , X ( 0 ) is the original feature matrix , Θ ( l ) are the trainable weights , and δ is the activation function . Â is the smoothing matrix that helps each node to smooth representations of neighboring nodes . As shown in Eq . 1 , each GCN layer contains two operations : feature aggregation ( smoothing ) and feature transformation . Figure 1 shows the framework of a two-layer GCN . The l-th layer in GCN firstly executes feature smoothing on the node embedding X ( l−1 ) . Then , the smoothed feature X̃ ( l−1 ) is transformed with trainable weights Θ ( l ) and activation function δ to generate new node embedding X ( l ) . Note that GCN will degrade to MLP if feature smoothing is removed from each layer . 2.3 GNN-BASED GRAPH REPRESENTATION LEARNING .. GAE ( Kipf & Welling , 2016b ) , the first and the most representative GNN-based graph embedding method , adopts an encoder to generate node embedding matrix Z with inputs Â and X . A simple inner product decoder is then used to reconstruct the adjacency matrix . The final training loss of GAE is the binary entropy loss between A′ and Ã , the reconstructed adjacency matrix and the original adjacency matrix with self loop added . A′ = sigmoid ( Z · ZT ) , L = ∑ 1≤i , j≤n −Ãi , j logA′i , j − ( 1− Ãi , j ) log ( 1−A′i , j ) ) . ( 2 ) Motivated by GAE , lots of GNN-based graph representation learning methods are proposed recently . MGAE ( Wang et al. , 2017 ) presents a denoising marginalized autoencoder that reconstructs the node feature matrix X. ARGA ( Pan et al. , 2018 ) adopts the adversarial learning strategy , and its generated node embeddings are forced to match a prior distribution . DAEGC ( Wang et al. , 2019 ) exploits side information to generate node embeddings in a self-supervised way . AGC ( Zhang et al. , 2019 ) proposes an improved filter matrix to better filter out the high-frequency noise . AGE ( Cui et al. , 2020 ) further improves AGC by using the similarity of embedding rather than the adjacency matrix to consider original node feature information . Compared with GNN-based graph representation learning methods that rely on the trainable parameters to learn node embeddings , our NAFS is training-free and thus enjoys higher efficiency and scalability . 3 OBSERVATION AND INSIGHT . In this section , we make a quantitative analysis on the over-smoothing issue at the node level and then provide some insights when designing NAFS on graphs . 3.1 FEATURE SMOOTHING IN DECOUPLED GNNS . Recently , many works ( Wu et al. , 2019 ; Zhu & Koniusz , 2021 ; Chen et al. , 2020 ; Zhang et al. , 2021 ) propose to decouple the feature smoothing and feature transformation in each GCN layer for scalable node classification . Concretely , they execute the feature smoothing operation in advance , and the smoothed features are then fed into a simple MLP to generate the final predicted node labels . Under this framework , the predictive node classification accuracy of these methods is comparable with or even higher than the one of coupled GNNs , and these works claim that the true power of GNNs lies in feature smoothing rather than feature transformation . We split the framework of these decoupled GNNs into two parts : feature smoothing and MLP training . Feature smoothing aims to combine the graph structural information and node features into better features for the subsequent MLP ; while MLP training only takes in the smoothed feature and is specially trained for a given task . As stated by previous decoupled GNNs ( Wu et al. , 2019 ; Zhu & Koniusz , 2021 ) , the true success of GNNs lies in feature smoothing rather than feature transformation . Correspondingly , we propose to remove feature transformation and preserve the key feature smoothing part alone for simple and scalable node representation . There is another branch of GNNs that also decouple the feature smoothing and feature transformation . The most representative method of this category is APPNP ( Klicpera et al. , 2018 ) . It first feeds the raw node features into an MLP to generate intermediate node embeddings ; then the personalized PageRank based propagation operations are performed on the node embeddings to produce final prediction results . However , compared with scalable decoupled GNNs mentioned in the previous paragraph , this branch of GNNs still have to recursively execute propagation operations in each training epoch , which makes it impossible to perform on large-scale graphs . In the remaining part of this paper , the terminology “ decoupled GNNs ” refers particularly to the scalable decoupled GNNs mentioned in the previous two paragraphs . 3.2 MEASURING SMOOTHING LEVEL . To capture deep graph structural information , a straightforward way is to simply stack multiple GNN layers . However , a large number of feature smoothing operations in a GNN model would lead to indistinguishable node embeddings , i.e. , the over-smoothing issue ( Li et al. , 2018 ) . Concretely , if we execute ÂX for infinite times , the node embeddings within the same connected component would reach a stationary state . When adopting Â = D̃r−1ÃD̃−r , Â∞ follows Â∞i , j = ( di + 1 ) r ( dj + 1 ) 1−r 2m+ n , ( 3 ) which shows that the influence from node vi to vj is only determined by their degrees . Under the extreme condition that r = 0 , all the nodes within one connected component have exactly the same representation , making it impossible to apply the node embeddings to subsequent tasks . Here we introduce a new metric , “ Over-smoothing Distance ” , to measure each node ’ s smoothing level . A smaller value indicates that the node is closer to the stationary state , i.e. , closer to over-smoothing . Definition 3.1 ( Over-smoothing Distance ) . The Over-smoothing Distance Di ( k ) parameterized by node i and smoothing step k is defined as Di ( k ) = Dis ( [ Â kX ] i , [ Â ∞X ] i ) , ( 4 ) where [ ÂkX ] i denotes the ith row of ÂkX , representing the representations of node vi after smoothing k times ; [ Â∞X ] i denotes the ith row of Â∞X , representing the stationary state of node vi ; Dis ( · ) is a distance function or a function positively relative with the difference , which can be implemented using Euclidean distance , the inverse of cosine similarity , etc .

Summary: The paper deals with (unsupervised) learning with graphs, specifically node-level tasks. Inspired by spectral GNNs, specifically Graph Convolutional Networks (GCN), the authors propose a simple neighborhood smoothing technique to capture the graph structure around each node in the given graph. Contrary to GNN, the proposed algorithm is parameter-free and hence scales better than their end-to-end trained GNN counterpart.                      Motivated by the problem of over-smoothing of GCN (Li et al., 2018), they propose the so-called "over-smoothing distance" measuring how close a node's feature is to be over-smoothed, which is simply the row-wise distance between $A^k X$ and $A^{\infty}X$ with regard to the Euclidian distance, see Def. 3.1. Based on this distance they define the smoothing weight matrix which is then used to weight the neighboring node features during neighborhood aggregation. The resulting features $\hat{X}$ are computed as $\hat{X} =  \sum^{K}_{k=0} W(k)\hat{A}X$, where $W(k)$ is the $k$th smoothing weight matrix.  The proposed architecture is evaluated on standard, old, small-scale (unsupervsied) link and node classification (Cora, Citeseer, PubMed) tasks showing somewhat classification performance while showing a considerable speedup in computation time.