Unifying Graph Convolutional Neural Networks and Label Propagation

Label Propagation ( LPA ) and Graph Convolutional Neural Networks ( GCN ) are both message passing algorithms on graphs . Both solve the task of node classification but LPA propagates node label information across the edges of the graph , while GCN propagates and transforms node feature information . However , while conceptually similar , theoretical relation between LPA and GCN has not yet been investigated . Here we study the relationship between LPA and GCN in terms of two aspects : ( 1 ) feature/label smoothing where we analyze how the feature/label of one node is spread over its neighbors ; And , ( 2 ) feature/label influence of how much the initial feature/label of one node influences the final feature/label of another node . Based on our theoretical analysis , we propose an end-to-end model that unifies GCN and LPA for node classification . In our unified model , edge weights are learnable , and the LPA serves as regularization to assist the GCN in learning proper edge weights that lead to improved classification performance . Our model can also be seen as learning attention weights based on node labels , which is more task-oriented than existing feature-based attention models . In a number of experiments on real-world graphs , our model shows superiority over state-of-the-art GCN-based methods in terms of node classification accuracy . 1 INTRODUCTION . Consider the problem of node classification in a graph , where the goal is to learn a mappingM : V → L from nodes V to labels L. Solution to this problem is widely applicable to various scenarios , e.g. , inferring income of users in a social network or classifying scientific articles in a citation network . Different from a generic machine learning problem where samples are independent from each other , nodes are connected by edges in the graph , which provide additional information and require more delicate modeling . To capture the graph information , researchers have mainly designed models on the assumption that labels and features vary smoothly over the edges of the graph . In particular , on the label side L , node labels are propagated and aggregated along edges in the graph , which is known as Label Propagation Algorithm ( LPA ) ( Zhu et al. , 2005 ; Zhou et al. , 2004 ; Zhang & Lee , 2007 ; Wang & Zhang , 2008 ; Karasuyama & Mamitsuka , 2013 ; Gong et al. , 2017 ; Liu et al. , 2019a ) ; On the node side V , node features are propagated along edges and transformed through neural network layers , which is known as Graph Convolutional Neural Networks ( GCN ) ( Kipf & Welling , 2017 ; Hamilton et al. , 2017 ; Li et al. , 2018 ; Xu et al. , 2018 ; Liao et al. , 2019 ; Xu et al. , 2019 ; Qu et al. , 2019 ) . GCN and LPA are related in that they propagate features and labels on the two sides of the mapping M , respectively . However , the relationship between GCN and LPA has not yet been investigated . Specifically , what is the theoretical relationship between GCN and LPA , and how can they be combined to develop a more accurate model for node classification in graphs ? Here we study the theoretical relationship between GCN and LPA from two viewpoints : ( 1 ) Feature/label smoothing , where we show that the intuition behind GCN/LPA is smoothing features/labels of nodes across the edges of the graph , i.e. , one node ’ s feature/label equals the weighted average of features/labels of its neighbors . We prove that if edge weights smooth the node features , they also smooth the node labels with guaranteed upper bound on the approximation error . And , ( 2 ) feature/label influence , where we quantify how much the initial feature/label of node vb influences the output feature/label of node va in GCN/LPA by studying the Jacobian/gradient of node vb with respect to node va , and then we also prove their quantitative relationship . Based on the above theoretical analysis , we propose a unified model GCN-LPA for node classification . We show that the key to improving the performance of GCN is to enable nodes within the same class/label to connect more strongly with each other by making edge weights/strengths trainable . Then we prove that increasing the strength of edges between the nodes of the same class is equivalent to increasing the accuracy of LPA ’ s predictions . Therefore , we can first learn the optimal edge weights by minimizing the loss of predictions in LPA , then plug the optimal edge weights into a GCN to learn node representations and do final classification . In GCN-LPA , we further combine the two steps together and train the whole model in an end-to-end fashion , where the LPA part serves as regularization to assist the GCN part in learning proper edge weights that benefit the separation of different classes . It is worth noticing that GCN-LPA can also be seen as learning attention weights for edges based on node label information , which requires less handcrafting and is more task-oriented than existing work that learns attention weights based on node feature similarity ( Veličković et al. , 2018 ; Thekumparampil et al. , 2018 ; Zhang et al. , 2018 ; Liu et al. , 2019b ) . Experiments on five datasets indicate that our model outperforms state-of-the-art methods in terms of classification accuracy . 2 UNIFYING GCN AND LPA . In this section , we first formulate the node classification problem and briefly introduce LPA and GCN . We then prove their relationship from the viewpoints of smoothing and influence . Based on our theoretical analysis , we then propose a unified model GCN-LPA , and we also analyze the reason why our model performs better than GCN in node classification . 2.1 PROBLEM FORMULATION AND PRELIMINARIES . We begin by describing the problem of node classification on graphs and introducing notation . Consider a graph G = ( V , A , X , Y ) , where V = { v1 , · · · , vn } is the set of nodes , A ∈ Rn×n is the adjacency matrix ( self-loops are included ) , X is the feature matrix of nodes and Y is labels of nodes . aij ( the ij-th entry of A ) is the weight of the edge connecting vi and vj . N ( v ) denotes the set of immediate neighbors of node v in graph G. Each node vi has a feature vector xi which is the i-th row of X , while only the first m nodes have labels y1 , · · · , ym from a label set L = { 1 , · · · , c } . The goal is to learn a mappingM : V → L and predict labels of unlabeled nodes . Label Propagation Algorithm . LPA assumes that two connected nodes are likely to have the same label , and thus it propagates labels iteratively along the edges . Let Y ( k ) = [ y ( k ) 1 , · · · , y ( k ) n ] > ∈ Rn×c be the soft label matrix in iteration k > 0 , in which the i-th row y ( k ) > i denotes the predicted label distribution for node vi in iteration k. When k = 0 , the initial label matrix Y ( 0 ) = [ y ( 0 ) 1 , · · · , y ( 0 ) n ] > consists of one-hot label indicator vectors y ( 0 ) i for i = 1 , · · · , m ( i.e. , labeled nodes ) or zero vectors otherwise ( i.e. , unlabeled nodes ) . Let D be the diagonal degree matrix for A with entries dii =∑ j aij . Then LPA ( Zhu et al. , 2005 ) in iteration k is formulated as the following two steps : Y ( k+1 ) = D−1A Y ( k ) , ( 1 ) y ( k+1 ) i = y ( 0 ) i , ∀ i ≤ m. ( 2 ) In Eq . ( 1 ) , all nodes propagate their labels to their neighbors according to weights of edges . Then in Eq . ( 2 ) , labels of all labeled nodes are reset to their initial values , because LPA wants to persist labels of nodes which are labeled so that unlabeled nodes do not overpower the labeled ones as the initial labels would otherwise fade away . Graph Convolutional Neural Network . GCN is a multi-layer feedforward neural network that propagates and transforms node features across the graph . The layer-wise propagation rule of GCN is X ( k+1 ) = σ ( D− 1 2AD− 1 2X ( k ) W ( k ) ) , where W ( k ) is trainable weight matrix in the k-th layer , σ ( · ) is an activation function such as ReLU , and X ( k ) = [ x ( k ) 1 , · · · , x ( k ) n ] > are the k-th layer node representations with X ( 0 ) = X . To align with the above LPA , we use D−1A as the normalized adjacency matrix instead of the symmetric one D− 1 2AD− 1 2 proposed by Kipf & Welling ( 2017 ) . Therefore , the feature propagation scheme of GCN in layer k is : X ( k+1 ) = σ ( D−1AX ( k ) W ( k ) ) . ( 3 ) Notice similarity between Eqs . ( 1 ) and ( 3 ) . Next we shall study and uncover the relationship between the two equations . 2.2 FEATURE SMOOTHING AND LABEL SMOOTHING . The intuition behind both LPA and GCN is smoothing ( Zhu et al. , 2003 ; Li et al. , 2018 ) : In LPA , the final label of a node is the weighted average of labels of its neighbors : y ( ∞ ) i = 1 dii ∑ j∈N ( i ) aijy ( ∞ ) j ; in GCN , the final node representation is also the weighted average of representations of its neighbors if we assume σ is identity function and W ( · ) are identity matrices : x ( ∞ ) i = 1 dii ∑ j∈N ( i ) aijx ( ∞ ) j . Next we show the relationship between feature smoothing and label smoothing : Theorem 1 ( Relationship between feature smoothing and label smoothing ) Suppose that the latent ground-truth mapping M : x → y from node features to node labels is differentiable and satisfies L-Lipschitz constraint , i.e. , |M ( x1 ) − M ( x2 ) | ≤ L‖x1 − x2‖2 for any x1 and x2 ( L is a constant ) . If the edge weights { aij } approximately smooth xi over its immediate neighbors with error i , i.e. , xi = 1dii ∑ j∈N ( i ) aijxj + i , then the edge weights { aij } also approximately smooth yi over its immediate neighbors with the following approximation error : ∣∣yi − 1dii ∑j∈N ( i ) aijyj∣∣ ≤ L‖ i‖2 + o ( maxj∈N ( i ) ( ‖xj − xi‖2 ) ) , where o ( α ) denotes a higher order infinitesimal than α . Proof of Theorem 1 is in Appendix A. Theorem 1 indicates that label smoothing is theoretically guaranteed by feature smoothing . Note that if we treat edge weights { aij } learnable , then feature smoothing ( i.e. , i → 0 ) can be directly achieved by keeping node features xi fixed while setting { aij } appropriately , without resorting to feature propagation in a multi-layer GCN . Therefore , a simple approach to exploit this theorem would be to learn { aij } by reconstructing node feature xi from its neighbors , then use the learned { aij } to reconstruct node labels yi ( Karasuyama & Mamitsuka , 2013 ) . This is equivalent to first minimizing the difference between X ( 0 ) and X ( 1 ) by learning D−1A in a one-layer GCN where σ and W ( 0 ) are identity , then plug the learned D−1A in LPA and run it for one iteration . As shown in Theorem 1 , the approximation error of labels is dominated by L‖ i‖2 . However , this error could be fairly large in practice because : ( 1 ) The number of immediate neighbors for a given node may be too small to reconstruct its features perfectly , especially in the case where node features are high-dimensional and sparse . For example , in a citation network where node features are one-hot bag-of-words vectors , the feature of one article can never be precisely reconstructed if none of its neighboring articles contains the specific word that appears in this article . As a result , ‖ i‖2 will be non-neglibible . This explains why it is beneficial to apply LPA and GCN for multiple iterations/layers in order to include information from farther away neighbors . ( 2 ) The ground-truth mapping M may not be sufficiently smooth due to the complex structure of latent manifold and possible noise , which fails to satisfy L-Lipschitz constraint . In other words , the constant L will be extremely large .

Summary: The paper proposed an unified model for Label Propagation (LPA) and Graph Convolutional Neural Networks (GCN). It is shown how it is possible to infer the relationship between LPA and GCN in terms of label or feature smoothing (how label/feature does propagate over the neighbors) and label or feature influence over the other nodes. The results are given in terms of two theorems (whose proofs are in an appendix) which essentially state that the total label influence of nodes with a particular label “l” on a specific node is proportional to the probability that node is labelled as “l” by LPA. In practice, LPA acts as a regularizer to learn transformation matrices and edge weights simultaneously in GCN. By means of a simple joint loss (eq.8), the regularized training show that transductive learning with the joint model surpasses GCN/GNN baselines.