Combining Label Propagation and Simple Models out-performs Graph Neural Networks

1 INTRODUCTION . Following the success of neural networks in computer vision and natural language processing , there are now a wide range of graph neural networks ( GNNs ) for making predictions involving relational data ( Battaglia et al. , 2018 ; Wu et al. , 2020 ) . These models have had much success and sit atop leaderboards such as the Open Graph Benchmark ( Hu et al. , 2020 ) . Often , the methodological developments for GNNs revolve around creating strictly more expressive architectures than basic variants such as the Graph Convolutional Network ( GCN ) ( Kipf & Welling , 2017 ) or GraphSAGE ( Hamilton et al. , 2017a ) ; examples include Graph Attention Networks ( Veličković et al. , 2018 ) , Graph Isomorphism Networks ( Xu et al. , 2018 ) , and various deep models ( Li et al. , 2019 ; Rong et al. , 2019 ; Chen et al. , 2020 ) . Many ideas for new GNN architectures are adapted from new architectures in models for language ( e.g. , attention ) or vision ( e.g. , deep CNNs ) with the hopes that success will translate to graphs . However , as these models become more complex , understanding their performance gains is a major challenge , and scaling them to large datasets is difficult . Here , we see how far we can get by combining much simpler models , with an emphasis on understanding where there are easy opportunities for performance improvements in graph learning , particularly transductive node classification . We propose a simple pipeline with three main parts ( Figure 1 ) : ( i ) a base prediction made with node features that ignores the graph structure ( e.g. , a shallow multi-layer perceptron or just a linear model ) ; ( ii ) a correction step , which propagates uncertainties from the training data across the graph to correct the base prediction ; and ( iii ) a smoothing of the predictions over the graph . Steps ( ii ) and ( iii ) are post-processing and implemented with classical methods for graph-based semi-supervised learning , namely , label propagation techniques ∗Equal contribution †Work done while at Cornell University ( Zhu , 2005 ) .1 With a few modifications and new deployment of these classic ideas , we achieve stateof-the-art performance on several node classification tasks , outperforming big GNN models . In our framework , the graph structure is not used to learn parameters ( which is done in step ( i ) ) but instead as a post-processing mechanism . This simplicity leads to models with orders of magnitude fewer parameters that take orders of magnitude less time to train and can easily scale to large graphs . We can also combine our ideas with state-of-the-art GNNs , although the performance gains are modest . A major source of our performance improvements is directly using labels for predictions . This idea is not new — early diffusion-based semi-supervised learning algorithms on graphs such as the spectral graph transducer ( Joachims , 2003 ) , Gaussian random field models ( Zhu et al. , 2003 ) , and and label spreading ( Zhou et al. , 2004 ) all use this idea . However , the motivation for these methods was semi-supervised learning on point cloud data , so the “ node features ” were used to construct the graph itself . Since then , these techniques have been used for learning on relational data consisting of a graph and some labels but no node features ( Koutra et al. , 2011 ; Gleich & Mahoney , 2015 ; Peel , 2017 ; Chin et al. , 2019 ) ; however , they have largely been ignored in the context of GNNs . ( That being said , we still find that even simple label propagation , which ignores features , does surprisingly well on a number of benchmarks . ) This provides motivation for combining two orthogonal sources of prediction power — one coming from the node features ( ignoring graph structure ) and one coming from using the known labels directly in predictions . Recent research connects GNNs to label propagation ( Wang & Leskovec , 2020 ; Jia & Benson , 2020 ; 2021 ) as well as Markov Random fields ( Qu et al. , 2019 ; Gao et al. , 2019 ) , and some techniques use ad hoc incorporation of label information in the features ( Shi et al. , 2020 ) . However , these approaches are usually still expensive to train , while we use label propagation in two understandable and low-cost ways . We start with a cheap “ base prediction ” from a model that uses only node features and ignores the graph structure . After , we use label propagation for error correction and then to smooth final predictions . These post-processing steps are based on the fact that errors and labels on connected nodes tend to be positively correlated . Assuming similarity between connected nodes is at the center of much network analysis and corresponds to homophily or assortative mixing ( McPherson et al. , 2001 ; Newman , 2003 ; Easley & Kleinberg , 2010 ) . In the semi-supervised learning literature , the analog is the smoothness or cluster assumption ( Chapelle et al. , 2003 ; Zhu , 2005 ) . The good performance of label propagation that we see across a wide variety of datasets suggests that these correlations hold on common benchmarks . 1One of the main methods that we use ( Zhou et al. , 2004 ) is often called label spreading . The term “ label propagation ” is used in a variety of contexts ( Zhu , 2005 ; Wang & Zhang , 2007 ; Raghavan et al. , 2007 ; Gleich & Mahoney , 2015 ) . The salient point for this paper is that we assume positive correlations on neighboring nodes and that the algorithms work by “ propagating ” information from one node to another . Overall , our methodology demonstrates that combining several simple ideas yields excellent performance in transductive node classification at a fraction of the cost , in terms of both model size ( i.e. , number of parameters ) and training time . For example , on the OGB-Products benchmark , we out-perform the current best-known GNN with more than two orders of magnitude fewer parameters and more than two orders of magnitude less training time . However , our goal is not to say that current graph learning methods are poor or inappropriate . Instead , we aim to highlight easier ways in which to improve prediction performance in graph learning and to better understand the source of performance gains . Our main finding is that more direct incorporation of labels into the learning algorithms is key . We hope that our approach spurs new ideas that can help in other graph learning tasks , such as inductive node classification , link prediction , and graph prediction . 1.1 ADDITIONAL RELATED WORK . The Approximate Personalized Propagation of Neural Predictions ( APPNP ) framework is most relevant to our work , as they also smooth base predictions ( Klicpera et al. , 2018 ) . However , they focus on integrating this smoothing into the training process so that their model can be trained end to end . Not only is this significantly more computationally expensive , it also prevents APPNP from incorporating label information at inference . Compared to APPNP , our framework produces more accurate predictions , is faster to train , and more easily scales to large datasets . That being said , APPNP can also be used without end-to-end training , which can make it faster but less accurate . Our framework also complements the Simplified Graph Convolution ( Wu et al. , 2019 ) and other algorithms designed to increase scalability ( Bojchevski et al. , 2020 ; Zeng et al. , 2019 ; Frasca et al. , 2020 ) . The primary focus of our approach , however , is using labels directly , and scalability is a byproduct . There is also prior work connecting GCNs and label propagation . Wang & Leskovec ( 2020 ) use label propagation as a pre-processing step to weight edges for GNNs , whereas we use label propagation as a post-processing step and avoid GNNs . Jia & Benson ( 2020 ; 2021 ) use label propagation with GNNs for regression tasks , and our error correction step adapts some of their ideas for the case of classification . Finally , there are several recent approaches that incorporate nonlinearity into label propagation methods to compete with GNNs and achieve scalability ( Eliav & Cohen , 2018 ; Ibrahim & Gleich , 2019 ; Tudisco et al. , 2021 ) , but these methods focus on settings of low label rates and do not use feature-based learning . 2 CORRECT AND SMOOTH ( C & S ) MODEL . We start with some notation . We assume that we have an undirected graph G = ( V , E ) , where there are n = |V | nodes with features on each node represented by a matrix X ∈ Rn×p . Let A be the adjacency matrix of the graph , D be the diagonal degree matrix , and S be the normalized adjacency matrix D−1/2AD−1/2 . For the prediction problem , the node set V is split into a disjoint set of unlabeled nodes U and labeled nodes L , which are subsets of the indices { 1 , . . . , n } . We will further split the labeled nodes into a training set Lt and validation set Lv . We represent the labels by a one-hot-encoding matrix Y ∈ Rn×c , where c is the number of classes ( i.e. , Yij = 1 if i ∈ L is known to be in class j , and 0 otherwise , where the ith row of Y is all zero if i ∈ U ) , Our problem is transductive node classification : assign each node j ∈ U a label in { 1 , . . . , c } , given G , X , and Y . Our approach starts with a simple base predictor on node features that does not rely on any learning over the graph . After , we perform two types of label propagation ( LP ) : one that corrects the base predictions by modeling correlated error and one that smooths the final prediction . We call the combination of these two methods Correct and Smooth ( C & S ; Figure 1 ) . The LPs are only postprocessing steps , and our pipeline is not trained end-to-end . Furthermore , the graph is only used in the post-processing steps ( and in a pre-processing step to augment the features X ) , but not for the base predictions . This makes training fast and scalable compared to standard GNN models . Moreover , we take advantage of both LP ( which performs fairly well on its own without features ) and the node features . We find that combining these complementary signals yields excellent predictions . 2.1 SIMPLE BASE PREDICTOR . To start , we use a simple base predictor that does not rely on the graph structure . More specifically , we train a model f to minimize ∑ i∈Lt ` ( f ( xi ) , yi ) , where xi is the ith row of X , yi is the ith row of Y , and ` is a loss function , For this paper , f is either a linear model or a shallow multi-layer perceptron ( MLP ) followed by a softmax , and ` is the cross-entropy loss . The validation set Lv is used to tune hyperparameters such as learning rates and the hidden layer dimensions for the MLP . From f , we get a base prediction Z ∈ Rn×c , where each row of Z is a probability distribution resulting from the softmax . Omitting the graph structure for these base predictions avoids most of the scalability issues with GNNs . In principle , though , we can use any base predictor for Z , including those based on GNNs , and we explore this in Section 3 . However , for our pipeline to be simple and scalable , we just use linear classifiers or MLPs with subsequent post-processing , which we describe next .

Summary: This paper presents C&S method that predicts node labels in the transductive semi-supervised node classification setting. C&S uses the three-stage-pipeline approach. First, label probabilities are predicted with simple and scalable classifiers such as MLP. Then, the predicted errors are diffused over graphs. Finally, the labels are further smoothened to give the final node label prediction. The authors demonstrate that their simple C&S approach beats many existing GNN approaches.