Combining Label Propagation and Simple Models out-performs Graph Neural Networks

1 INTRODUCTION . Following the success of neural networks in computer vision and natural language processing , there are now a wide range of graph neural networks ( GNNs ) for making predictions involving relational data ( Battaglia et al. , 2018 ; Wu et al. , 2020 ) . These models have had much success and sit atop leaderboards such as the Open Graph Benchmark ( Hu et al. , 2020 ) . Often , the methodological developments for GNNs revolve around creating strictly more expressive architectures than basic variants such as the Graph Convolutional Network ( GCN ) ( Kipf & Welling , 2017 ) or GraphSAGE ( Hamilton et al. , 2017a ) ; examples include Graph Attention Networks ( Veličković et al. , 2018 ) , Graph Isomorphism Networks ( Xu et al. , 2018 ) , and various deep models ( Li et al. , 2019 ; Rong et al. , 2019 ; Chen et al. , 2020 ) . Many ideas for new GNN architectures are adapted from new architectures in models for language ( e.g. , attention ) or vision ( e.g. , deep CNNs ) with the hopes that success will translate to graphs . However , as these models become more complex , understanding their performance gains is a major challenge , and scaling them to large datasets is difficult . Here , we see how far we can get by combining much simpler models , with an emphasis on understanding where there are easy opportunities for performance improvements in graph learning , particularly transductive node classification . We propose a simple pipeline with three main parts ( Figure 1 ) : ( i ) a base prediction made with node features that ignores the graph structure ( e.g. , a shallow multi-layer perceptron or just a linear model ) ; ( ii ) a correction step , which propagates uncertainties from the training data across the graph to correct the base prediction ; and ( iii ) a smoothing of the predictions over the graph . Steps ( ii ) and ( iii ) are post-processing and implemented with classical methods for graph-based semi-supervised learning , namely , label propagation techniques ∗Equal contribution †Work done while at Cornell University ( Zhu , 2005 ) .1 With a few modifications and new deployment of these classic ideas , we achieve stateof-the-art performance on several node classification tasks , outperforming big GNN models . In our framework , the graph structure is not used to learn parameters ( which is done in step ( i ) ) but instead as a post-processing mechanism . This simplicity leads to models with orders of magnitude fewer parameters that take orders of magnitude less time to train and can easily scale to large graphs . We can also combine our ideas with state-of-the-art GNNs , although the performance gains are modest . A major source of our performance improvements is directly using labels for predictions . This idea is not new — early diffusion-based semi-supervised learning algorithms on graphs such as the spectral graph transducer ( Joachims , 2003 ) , Gaussian random field models ( Zhu et al. , 2003 ) , and and label spreading ( Zhou et al. , 2004 ) all use this idea . However , the motivation for these methods was semi-supervised learning on point cloud data , so the “ node features ” were used to construct the graph itself . Since then , these techniques have been used for learning on relational data consisting of a graph and some labels but no node features ( Koutra et al. , 2011 ; Gleich & Mahoney , 2015 ; Peel , 2017 ; Chin et al. , 2019 ) ; however , they have largely been ignored in the context of GNNs . ( That being said , we still find that even simple label propagation , which ignores features , does surprisingly well on a number of benchmarks . ) This provides motivation for combining two orthogonal sources of prediction power — one coming from the node features ( ignoring graph structure ) and one coming from using the known labels directly in predictions . Recent research connects GNNs to label propagation ( Wang & Leskovec , 2020 ; Jia & Benson , 2020 ; 2021 ) as well as Markov Random fields ( Qu et al. , 2019 ; Gao et al. , 2019 ) , and some techniques use ad hoc incorporation of label information in the features ( Shi et al. , 2020 ) . However , these approaches are usually still expensive to train , while we use label propagation in two understandable and low-cost ways . We start with a cheap “ base prediction ” from a model that uses only node features and ignores the graph structure . After , we use label propagation for error correction and then to smooth final predictions . These post-processing steps are based on the fact that errors and labels on connected nodes tend to be positively correlated . Assuming similarity between connected nodes is at the center of much network analysis and corresponds to homophily or assortative mixing ( McPherson et al. , 2001 ; Newman , 2003 ; Easley & Kleinberg , 2010 ) . In the semi-supervised learning literature , the analog is the smoothness or cluster assumption ( Chapelle et al. , 2003 ; Zhu , 2005 ) . The good performance of label propagation that we see across a wide variety of datasets suggests that these correlations hold on common benchmarks . 1One of the main methods that we use ( Zhou et al. , 2004 ) is often called label spreading . The term “ label propagation ” is used in a variety of contexts ( Zhu , 2005 ; Wang & Zhang , 2007 ; Raghavan et al. , 2007 ; Gleich & Mahoney , 2015 ) . The salient point for this paper is that we assume positive correlations on neighboring nodes and that the algorithms work by “ propagating ” information from one node to another . Overall , our methodology demonstrates that combining several simple ideas yields excellent performance in transductive node classification at a fraction of the cost , in terms of both model size ( i.e. , number of parameters ) and training time . For example , on the OGB-Products benchmark , we out-perform the current best-known GNN with more than two orders of magnitude fewer parameters and more than two orders of magnitude less training time . However , our goal is not to say that current graph learning methods are poor or inappropriate . Instead , we aim to highlight easier ways in which to improve prediction performance in graph learning and to better understand the source of performance gains . Our main finding is that more direct incorporation of labels into the learning algorithms is key . We hope that our approach spurs new ideas that can help in other graph learning tasks , such as inductive node classification , link prediction , and graph prediction . 1.1 ADDITIONAL RELATED WORK . The Approximate Personalized Propagation of Neural Predictions ( APPNP ) framework is most relevant to our work , as they also smooth base predictions ( Klicpera et al. , 2018 ) . However , they focus on integrating this smoothing into the training process so that their model can be trained end to end . Not only is this significantly more computationally expensive , it also prevents APPNP from incorporating label information at inference . Compared to APPNP , our framework produces more accurate predictions , is faster to train , and more easily scales to large datasets . That being said , APPNP can also be used without end-to-end training , which can make it faster but less accurate . Our framework also complements the Simplified Graph Convolution ( Wu et al. , 2019 ) and other algorithms designed to increase scalability ( Bojchevski et al. , 2020 ; Zeng et al. , 2019 ; Frasca et al. , 2020 ) . The primary focus of our approach , however , is using labels directly , and scalability is a byproduct . There is also prior work connecting GCNs and label propagation . Wang & Leskovec ( 2020 ) use label propagation as a pre-processing step to weight edges for GNNs , whereas we use label propagation as a post-processing step and avoid GNNs . Jia & Benson ( 2020 ; 2021 ) use label propagation with GNNs for regression tasks , and our error correction step adapts some of their ideas for the case of classification . Finally , there are several recent approaches that incorporate nonlinearity into label propagation methods to compete with GNNs and achieve scalability ( Eliav & Cohen , 2018 ; Ibrahim & Gleich , 2019 ; Tudisco et al. , 2021 ) , but these methods focus on settings of low label rates and do not use feature-based learning . 2 CORRECT AND SMOOTH ( C & S ) MODEL . We start with some notation . We assume that we have an undirected graph G = ( V , E ) , where there are n = |V | nodes with features on each node represented by a matrix X ∈ Rn×p . Let A be the adjacency matrix of the graph , D be the diagonal degree matrix , and S be the normalized adjacency matrix D−1/2AD−1/2 . For the prediction problem , the node set V is split into a disjoint set of unlabeled nodes U and labeled nodes L , which are subsets of the indices { 1 , . . . , n } . We will further split the labeled nodes into a training set Lt and validation set Lv . We represent the labels by a one-hot-encoding matrix Y ∈ Rn×c , where c is the number of classes ( i.e. , Yij = 1 if i ∈ L is known to be in class j , and 0 otherwise , where the ith row of Y is all zero if i ∈ U ) , Our problem is transductive node classification : assign each node j ∈ U a label in { 1 , . . . , c } , given G , X , and Y . Our approach starts with a simple base predictor on node features that does not rely on any learning over the graph . After , we perform two types of label propagation ( LP ) : one that corrects the base predictions by modeling correlated error and one that smooths the final prediction . We call the combination of these two methods Correct and Smooth ( C & S ; Figure 1 ) . The LPs are only postprocessing steps , and our pipeline is not trained end-to-end . Furthermore , the graph is only used in the post-processing steps ( and in a pre-processing step to augment the features X ) , but not for the base predictions . This makes training fast and scalable compared to standard GNN models . Moreover , we take advantage of both LP ( which performs fairly well on its own without features ) and the node features . We find that combining these complementary signals yields excellent predictions . 2.1 SIMPLE BASE PREDICTOR . To start , we use a simple base predictor that does not rely on the graph structure . More specifically , we train a model f to minimize ∑ i∈Lt ` ( f ( xi ) , yi ) , where xi is the ith row of X , yi is the ith row of Y , and ` is a loss function , For this paper , f is either a linear model or a shallow multi-layer perceptron ( MLP ) followed by a softmax , and ` is the cross-entropy loss . The validation set Lv is used to tune hyperparameters such as learning rates and the hidden layer dimensions for the MLP . From f , we get a base prediction Z ∈ Rn×c , where each row of Z is a probability distribution resulting from the softmax . Omitting the graph structure for these base predictions avoids most of the scalability issues with GNNs . In principle , though , we can use any base predictor for Z , including those based on GNNs , and we explore this in Section 3 . However , for our pipeline to be simple and scalable , we just use linear classifiers or MLPs with subsequent post-processing , which we describe next .

Summary: This paper shows modified label propagation can perform better than GCN. The idea is as follows: it first uses MLP on node features to get the initial labels, and then use two steps--correction and smoothness to postprocessing the labels. And this postprocessing is based on the traditional label propagation algorithm. It shows that this simple method matches GCN performances on various datasets.