Unifying Graph Convolutional Neural Networks and Label Propagation

Abstract: Label Propagation (LPA) and Graph Convolutional Neural Networks (GCN) are both message passing algorithms on graphs. Both solve the task of node classification but LPA propagates node label information across the edges of the graph, while GCN propagates and transforms node feature information. However, while conceptually similar, theoretical relation between LPA and GCN has not yet been investigated. Here we study the relationship between LPA and GCN in terms of two aspects: (1) feature/label smoothing where we analyze how the feature/label of one node are spread over its neighbors; And, (2) feature/label influence of how much the initial feature/label of one node influences the final feature/label of another node. Based on our theoretical analysis, we propose an end-to-end model that unifies GCN and LPA for node classification. In our unified model, edge weights are learnable, and the LPA serves as regularization to assist the GCN in learning proper edge weights that lead to improved classification performance. Our model can also be seen as learning attention weights based on node labels, which is more task-oriented than existing feature-based attention models. In a number of experiments on real-world graphs, our model shows superiority over state-of-the-art GCN-based methods in terms of node classification accuracy.
      

Reviews and author responses:

We appreciate reviewer’s helpful and detailed feedback. The updates in our paper are marked in red. 


1. Adding an experiment showing how much the LPA impacts the results.

We appreciate reviewer’s suggestion but in fact we have already performed it. In particular, in Figures 2 and 3 we vary the number of LPA iterations and the weight of LPA loss term $\lambda$, which reveals the impact of LPA on model performance. However, this point was not well explained in the paper and we made changes where we added an experiment in which we vary the ratio of labeled nodes in LPA, and the result is shown in new Table 3 in the revised paper.


We thank the reviewer for helpful and detailed feedback. Reviewer makes a number of helpful comments, which we addressed the feedback and included further clarifications in the paper to make the paper clearer and more understandable. The updates in our paper are marked in red.


1. The proof of Theorem 2 is not clear.

Thank you for raising this point. We made changes to our paper and have expanded the proof of Theorem 2 by adding more explanations, so that the proof is easier to follow. We also added Figure 5 as an illustrating example for the proof of Lemma 2.


2. Why “y” has to be reset at each iteration?

The reset step is part of the standard Label Propagation Algorithm (LPA), which was proposed by Zhu et al. 2005. The reason why this is a good idea is that LPA wants to persist labels of nodes which are labeled so that unlabeled nodes don’t overpower the labeled ones as the initial labels would otherwise fade away. The reviewer makes a good point and we have updated the paper (introduction of LPA in Section 2.1) to make this point clearer.


3. The improvement brought by the framework is marginal, even if it seems general.

The performance of our method exceeds almost all state of the art baselines on all datasets. Since the baselines like JK-Net are strong, and the accuracy results of baselines on most datasets are already very high (~0.9 or even higher), it is  not possible to surpass the baselines on these datasets by a very large margin. However, the improvement of our method is statistically significant and shows great results on standard benchmarks commonly used by the community.


4. The training is transductive.

The reviewer is right that we implemented our method based on the code of Kipf’s GCN, and therefore it is transductive. However, our method can be easily generalized to inductive case if we implement it in a way similar to GraphSAGE. Therefore, being transductive or inductive does not affect the novelty and impact of this work. We thank the reviewer for the comment and we have updated the manuscript to make it clearer that our results apply to both transductive as well as inductive settings.


We really appreciate reviewer’s helpful and detailed feedback. Our response to raised questions is below. The updates in our paper are marked in red. Overall, we observe that reviewed makes a number of questions that can be easily clarified.


1. What is the meaning of a L2 norms between features?

In Theorem 1, the term $||x_1 - x_2||_2$ is the Euclidean distance between two vectors $x_1$ and $x_2$, which are raw feature vectors for node $v_1$ and $v_2$ in the graph. The inequality $|M(x_1) - M(x_2)| \leq L ||x_1 - x_2||_2$ in Theorem 1 is a standard $L$-Lipschitz constraint, meaning that the function $M$ is not too wiggly (i.e., if you move $x$ for distance $1$ then the change of $M(x)$ will not exceed $L$). Reviewer can learn more about the $L$-Lipschitz constraint in https://en.wikipedia.org/wiki/Lipschitz_continuity, which provides a good overview of the topic.


2. How do you relate the claim of bag-of-words features used in experiments to your theoretical analysis?

Our theoretical analysis is applicable to any form of node features, as long as node features can be represented as a vector of a fixed length. This is also by far the most common way to represent features/attributes of nodes in a graph.

As reviewer mentioned, in the datasets used in our experiments, the node features are bag-of-words features. We are a bit unsure why reviewer thinks this conflicts with theoretical analysis. Reviewer should note that Euclidean distance is still well defined on sparse multi-hot vectors, that is vectors where many components take a zero value and other components take non-zero values.


3. How do you take a derivative with respect to a bag of words?

By bag-of-words features we mean a multi-hot vector encoding of which words appear in the document. For example, bag of words is a vector $[1, 0, 0, …, 1, 0]$  which the length is the vocabulary size and position/dimension $i$ indicates whether word $i$ appears in the given document or not.

Therefore, the derivatives are taken w.r.t. to a vector, as we defined in Definition 1. And the derivative of a vector $x$ w.r.t. another vector $y$ is then defined as a matrix, in which element $(i, j)$ is the derivative of $x[i]$ w.r.t. $y[j]$. In mathematics this is also known as the Jacobian matrix.


4. The constraints on A* is not specified in Eq. 5 (Eq. 5 is now Eq. 6 in the revised paper).

This is a good point and we thank the reviewer for making it. We have added a footnote in our paper (above Eq. 6) further clarifying this point.


5. The definition of y_hat is not specified in Eq. 5 (Eq. 5 is now Eq. 6 in the revised paper).

y_hat is already defined in the last line of Theorem 3. To make the presentation clearer, we also modified our paper and introduced the definition of y_hat below Eq. 6. Please note that we added a superscript “lpa” to y_hat in Eq. 6 in the revised paper.


The paper proposed an unified model for Label Propagation (LPA) and Graph Convolutional Neural Networks (GCN). It is shown how it is possible to infer the relationship between LPA and GCN in terms of label or feature smoothing (how label/feature does propagate over the neighbors) and label or feature influence over the other nodes. The results are given in terms of two theorems (whose proofs are in an appendix) which essentially state that the total label influence of nodes with a particular label “l” on a specific node is proportional to the probability that node is labelled as “l” by LPA. In practice, LPA acts as a regularizer to learn transformation matrices and edge weights simultaneously in GCN. By means of a simple joint loss (eq.8), the regularized training show that transductive learning with the joint model surpasses GCN/GNN baselines. 
While the proof of the first theorem is reasonable and seems correct (Taylor+ Schwarz inequality + L-Lip), the second left me a little puzzled, in particular wrt eq 16 and 17. Is it possible to add a graphical explanation or idea of the proof? Why “y” has to be reset at each iteration?
Unfortunately, the improvement brought by the framework is marginal, even if it seems general. 
The other limitation is that we have transductive training, but the authors are well aware about this.


This paper introduces a unified model which combines label propagation algorithm (LPA) and graph convolutional networks (GCNs) for node classification. The motivation of this combination is supported by two analysis on the feature/label smoothing and feature/label influence. The proposed GCN-LPA framework utilizeds LPA to adjust the edge weight A* through the label information. Then, this edge weight A* is used to transfer the knowledge from label information to feature information for enhancing the representation learning in GCN. An end-to-end  solution is proposed by treating the LPA process as regularization. Overall, the idea of unifying GCNs and LAP in an end-to-end fashion is interesting. 

One major concern is that from the experiment, it is unclear how much the LPA impacts the node classification. It will be more convincing if the performance comparison under different percentage of labeled samples (during LPA) is provided. 


This paper connects graph convolutional networks with label propagation. There are a numbder of issues that need to be solved before possible publication.
The theoretical part (secion 2) is hard to follow. For example, the authors introduce in 2.1 a mapping M from vertices to labels. Then in Theorem 1, this mapping M indeed maps features of the vertices to labels. But then waht is the meaning of a L2 norms between features? At this point, I had a look in the experiment section to see what features are considered in practice. In section 4.1, it is written for the citation networks: 'each node has a sparse bag-of-words feature vector' or in the coauthor networks: 'Node features represent paper keywords for each author's paper'. How do you relate these claims to your theoretical analysis? Things get even worse in section 2.3, where derivatives with respect to initial feature vector are taken. How do you take a derivative with respect to a bag of words?
Equation (5) is not clear at all, we need to read the end of this section to understand that A^* is constrained to have the same support as the adjacency graph and computed as a function of the node features. The authors shuold also define clearly y_hat in (5).

Meta-review: The authors attempt to unify graph convolutional networks and label propagation and propose a model that unifies them. The reviewers liked the idea but felt that more extensive experiments are needed. The impact of labels needs to be specially studied more in-depth.

Decision: reject