Unifying Graph Convolutional Neural Networks and Label Propagation

Label Propagation ( LPA ) and Graph Convolutional Neural Networks ( GCN ) are both message passing algorithms on graphs . Both solve the task of node classification but LPA propagates node label information across the edges of the graph , while GCN propagates and transforms node feature information . However , while conceptually similar , theoretical relation between LPA and GCN has not yet been investigated . Here we study the relationship between LPA and GCN in terms of two aspects : ( 1 ) feature/label smoothing where we analyze how the feature/label of one node is spread over its neighbors ; And , ( 2 ) feature/label influence of how much the initial feature/label of one node influences the final feature/label of another node . Based on our theoretical analysis , we propose an end-to-end model that unifies GCN and LPA for node classification . In our unified model , edge weights are learnable , and the LPA serves as regularization to assist the GCN in learning proper edge weights that lead to improved classification performance . Our model can also be seen as learning attention weights based on node labels , which is more task-oriented than existing feature-based attention models . In a number of experiments on real-world graphs , our model shows superiority over state-of-the-art GCN-based methods in terms of node classification accuracy . 1 INTRODUCTION . Consider the problem of node classification in a graph , where the goal is to learn a mappingM : V → L from nodes V to labels L. Solution to this problem is widely applicable to various scenarios , e.g. , inferring income of users in a social network or classifying scientific articles in a citation network . Different from a generic machine learning problem where samples are independent from each other , nodes are connected by edges in the graph , which provide additional information and require more delicate modeling . To capture the graph information , researchers have mainly designed models on the assumption that labels and features vary smoothly over the edges of the graph . In particular , on the label side L , node labels are propagated and aggregated along edges in the graph , which is known as Label Propagation Algorithm ( LPA ) ( Zhu et al. , 2005 ; Zhou et al. , 2004 ; Zhang & Lee , 2007 ; Wang & Zhang , 2008 ; Karasuyama & Mamitsuka , 2013 ; Gong et al. , 2017 ; Liu et al. , 2019a ) ; On the node side V , node features are propagated along edges and transformed through neural network layers , which is known as Graph Convolutional Neural Networks ( GCN ) ( Kipf & Welling , 2017 ; Hamilton et al. , 2017 ; Li et al. , 2018 ; Xu et al. , 2018 ; Liao et al. , 2019 ; Xu et al. , 2019 ; Qu et al. , 2019 ) . GCN and LPA are related in that they propagate features and labels on the two sides of the mapping M , respectively . However , the relationship between GCN and LPA has not yet been investigated . Specifically , what is the theoretical relationship between GCN and LPA , and how can they be combined to develop a more accurate model for node classification in graphs ? Here we study the theoretical relationship between GCN and LPA from two viewpoints : ( 1 ) Feature/label smoothing , where we show that the intuition behind GCN/LPA is smoothing features/labels of nodes across the edges of the graph , i.e. , one node ’ s feature/label equals the weighted average of features/labels of its neighbors . We prove that if edge weights smooth the node features , they also smooth the node labels with guaranteed upper bound on the approximation error . And , ( 2 ) feature/label influence , where we quantify how much the initial feature/label of node vb influences the output feature/label of node va in GCN/LPA by studying the Jacobian/gradient of node vb with respect to node va , and then we also prove their quantitative relationship . Based on the above theoretical analysis , we propose a unified model GCN-LPA for node classification . We show that the key to improving the performance of GCN is to enable nodes within the same class/label to connect more strongly with each other by making edge weights/strengths trainable . Then we prove that increasing the strength of edges between the nodes of the same class is equivalent to increasing the accuracy of LPA ’ s predictions . Therefore , we can first learn the optimal edge weights by minimizing the loss of predictions in LPA , then plug the optimal edge weights into a GCN to learn node representations and do final classification . In GCN-LPA , we further combine the two steps together and train the whole model in an end-to-end fashion , where the LPA part serves as regularization to assist the GCN part in learning proper edge weights that benefit the separation of different classes . It is worth noticing that GCN-LPA can also be seen as learning attention weights for edges based on node label information , which requires less handcrafting and is more task-oriented than existing work that learns attention weights based on node feature similarity ( Veličković et al. , 2018 ; Thekumparampil et al. , 2018 ; Zhang et al. , 2018 ; Liu et al. , 2019b ) . Experiments on five datasets indicate that our model outperforms state-of-the-art methods in terms of classification accuracy . 2 UNIFYING GCN AND LPA . In this section , we first formulate the node classification problem and briefly introduce LPA and GCN . We then prove their relationship from the viewpoints of smoothing and influence . Based on our theoretical analysis , we then propose a unified model GCN-LPA , and we also analyze the reason why our model performs better than GCN in node classification . 2.1 PROBLEM FORMULATION AND PRELIMINARIES . We begin by describing the problem of node classification on graphs and introducing notation . Consider a graph G = ( V , A , X , Y ) , where V = { v1 , · · · , vn } is the set of nodes , A ∈ Rn×n is the adjacency matrix ( self-loops are included ) , X is the feature matrix of nodes and Y is labels of nodes . aij ( the ij-th entry of A ) is the weight of the edge connecting vi and vj . N ( v ) denotes the set of immediate neighbors of node v in graph G. Each node vi has a feature vector xi which is the i-th row of X , while only the first m nodes have labels y1 , · · · , ym from a label set L = { 1 , · · · , c } . The goal is to learn a mappingM : V → L and predict labels of unlabeled nodes . Label Propagation Algorithm . LPA assumes that two connected nodes are likely to have the same label , and thus it propagates labels iteratively along the edges . Let Y ( k ) = [ y ( k ) 1 , · · · , y ( k ) n ] > ∈ Rn×c be the soft label matrix in iteration k > 0 , in which the i-th row y ( k ) > i denotes the predicted label distribution for node vi in iteration k. When k = 0 , the initial label matrix Y ( 0 ) = [ y ( 0 ) 1 , · · · , y ( 0 ) n ] > consists of one-hot label indicator vectors y ( 0 ) i for i = 1 , · · · , m ( i.e. , labeled nodes ) or zero vectors otherwise ( i.e. , unlabeled nodes ) . Let D be the diagonal degree matrix for A with entries dii =∑ j aij . Then LPA ( Zhu et al. , 2005 ) in iteration k is formulated as the following two steps : Y ( k+1 ) = D−1A Y ( k ) , ( 1 ) y ( k+1 ) i = y ( 0 ) i , ∀ i ≤ m. ( 2 ) In Eq . ( 1 ) , all nodes propagate their labels to their neighbors according to weights of edges . Then in Eq . ( 2 ) , labels of all labeled nodes are reset to their initial values , because LPA wants to persist labels of nodes which are labeled so that unlabeled nodes do not overpower the labeled ones as the initial labels would otherwise fade away . Graph Convolutional Neural Network . GCN is a multi-layer feedforward neural network that propagates and transforms node features across the graph . The layer-wise propagation rule of GCN is X ( k+1 ) = σ ( D− 1 2AD− 1 2X ( k ) W ( k ) ) , where W ( k ) is trainable weight matrix in the k-th layer , σ ( · ) is an activation function such as ReLU , and X ( k ) = [ x ( k ) 1 , · · · , x ( k ) n ] > are the k-th layer node representations with X ( 0 ) = X . To align with the above LPA , we use D−1A as the normalized adjacency matrix instead of the symmetric one D− 1 2AD− 1 2 proposed by Kipf & Welling ( 2017 ) . Therefore , the feature propagation scheme of GCN in layer k is : X ( k+1 ) = σ ( D−1AX ( k ) W ( k ) ) . ( 3 ) Notice similarity between Eqs . ( 1 ) and ( 3 ) . Next we shall study and uncover the relationship between the two equations . 2.2 FEATURE SMOOTHING AND LABEL SMOOTHING . The intuition behind both LPA and GCN is smoothing ( Zhu et al. , 2003 ; Li et al. , 2018 ) : In LPA , the final label of a node is the weighted average of labels of its neighbors : y ( ∞ ) i = 1 dii ∑ j∈N ( i ) aijy ( ∞ ) j ; in GCN , the final node representation is also the weighted average of representations of its neighbors if we assume σ is identity function and W ( · ) are identity matrices : x ( ∞ ) i = 1 dii ∑ j∈N ( i ) aijx ( ∞ ) j . Next we show the relationship between feature smoothing and label smoothing : Theorem 1 ( Relationship between feature smoothing and label smoothing ) Suppose that the latent ground-truth mapping M : x → y from node features to node labels is differentiable and satisfies L-Lipschitz constraint , i.e. , |M ( x1 ) − M ( x2 ) | ≤ L‖x1 − x2‖2 for any x1 and x2 ( L is a constant ) . If the edge weights { aij } approximately smooth xi over its immediate neighbors with error i , i.e. , xi = 1dii ∑ j∈N ( i ) aijxj + i , then the edge weights { aij } also approximately smooth yi over its immediate neighbors with the following approximation error : ∣∣yi − 1dii ∑j∈N ( i ) aijyj∣∣ ≤ L‖ i‖2 + o ( maxj∈N ( i ) ( ‖xj − xi‖2 ) ) , where o ( α ) denotes a higher order infinitesimal than α . Proof of Theorem 1 is in Appendix A. Theorem 1 indicates that label smoothing is theoretically guaranteed by feature smoothing . Note that if we treat edge weights { aij } learnable , then feature smoothing ( i.e. , i → 0 ) can be directly achieved by keeping node features xi fixed while setting { aij } appropriately , without resorting to feature propagation in a multi-layer GCN . Therefore , a simple approach to exploit this theorem would be to learn { aij } by reconstructing node feature xi from its neighbors , then use the learned { aij } to reconstruct node labels yi ( Karasuyama & Mamitsuka , 2013 ) . This is equivalent to first minimizing the difference between X ( 0 ) and X ( 1 ) by learning D−1A in a one-layer GCN where σ and W ( 0 ) are identity , then plug the learned D−1A in LPA and run it for one iteration . As shown in Theorem 1 , the approximation error of labels is dominated by L‖ i‖2 . However , this error could be fairly large in practice because : ( 1 ) The number of immediate neighbors for a given node may be too small to reconstruct its features perfectly , especially in the case where node features are high-dimensional and sparse . For example , in a citation network where node features are one-hot bag-of-words vectors , the feature of one article can never be precisely reconstructed if none of its neighboring articles contains the specific word that appears in this article . As a result , ‖ i‖2 will be non-neglibible . This explains why it is beneficial to apply LPA and GCN for multiple iterations/layers in order to include information from farther away neighbors . ( 2 ) The ground-truth mapping M may not be sufficiently smooth due to the complex structure of latent manifold and possible noise , which fails to satisfy L-Lipschitz constraint . In other words , the constant L will be extremely large .

Summary: This paper introduces a unified model which combines label propagation algorithm (LPA) and graph convolutional networks (GCNs) for node classification. The motivation of this combination is supported by two analysis on the feature/label smoothing and feature/label influence. The proposed GCN-LPA framework utilizeds LPA to adjust the edge weight A* through the label information. Then, this edge weight A* is used to transfer the knowledge from label information to feature information for enhancing the representation learning in GCN. An end-to-end  solution is proposed by treating the LPA process as regularization. Overall, the idea of unifying GCNs and LAP in an end-to-end fashion is interesting.