DEEP GRAPH TREE NETWORKS

Abstract: We propose Graph Tree Networks (GTree), a self-interpretive deep graph neural network architecture which originates from the tree representation of the graphs. In the tree representation, each node forms its own tree where the node itself is the root node and all its neighbors up to hop-k are the subnodes. Under the tree representation, the message propagates upward from the leaf nodes to the root node naturally and straightforwardly to update the root node's hidden features. This message passing (or neighborhood aggregation) scheme is essentially different from that in the vanilla GCN, GAT and many of their derivatives, and is demonstrated experimentally a superior message passing scheme. Models adopting this scheme has the capability of going deep. Two scalable graph learning models are proposed within this GTree network architecture - Graph Tree Convolution Network (GTCN) and Graph Tree Attention Network (GTAN), with demonstrated state-of-the-art performances on several benchmark datasets. The deep capability is also demonstrated for both models.

Reviews and author responses:

 3. In terms of contribution, we propose a general message passing rule in our GTreeNet which may open new research opportunities by exploring combinations of different aggregation functions and transformations. Our GTCN and GTAN models achieve state-of-the-art performance on several popular benchmark datasets. They have potentials to be extended to work with heterogeneous graphs.

Hope our explanation addresses your concerns and changes your mind! Please let us know if you have further questions or concerns.

Thank you!

 Thank you for your response. We will elaborate on each of your comment.

1. We do not claim the tree structure as a contribution. We claim the Graph Tree Network derived from the tree representation as one of our contributions. We assume that a node $u$ at any k-hop in the tree **preserves its initial information prior to receiving information from its child nodes**. With this assumption, we derive Graph Tree Networks (GTreeNets) with the following general message passing rule
$$
h_u^k = f_k (\text{aggregate}(\phi(x_u), \\{h_v^{k+1}, \forall v \in N_u \\}))
$$
That is, at k-th hop, a node $u$'s hidden feature is updated by the aggregation of its neighbor's hidden features and **its own initial information**.
In Jure Leskovec's GCN tutorial, every node's computation graph was presented as a tree though, GCN is not derived from the tree representation. In GCN, a node $u$'s hidden feature is updated by the aggregation of its neighbor's hidden features and **its own hidden features from last propagation layer**. This message passing rule cannot be interpreted by the tree representation without further modification of the tree structure. We like your suggestion of illustrating this difference with two figures. We will add this comparison in our updated paper. We will try our best to explain the difference between these two networks with words at this moment. If you insist to derive GCN from tree representation, the classical GCN can be derived from the tree with added self loops and the assumption that all nodes do not preserve any information prior to receiving information from its child nodes. This modification and assumption do not seem intuitive to us. To see this difference in networks, in GCN, there will be a connection from node $u$ in layer $k$ to node $u$ in layer $k+1$ for all node $u$, i.e. the output hidden feature of a node $u$ from the $k^{th}$ layer is always an input to the node $u$ at the $(k+1)^{th}$ layer; while in our GTreeNet, we don't have such connection, instead we have the output $z_u$ from the original transformation layer as input for node $u$ for all following propagation layers. 

2. Our GTCN model is different from APPNP. 
    * First, the aggregation information used by APPNP and our GTCN are different. In APPNP, hidden features of node $u$'s neighbors, hidden features of node $u$ from last propagation layer and original features of node $u$ are aggregated together to update node $u$'s hidden features. Our GTCN model and GTAN model do not use hidden features of node $u$ from last propagation layer to update its own hidden features in the aggregation. The reason is explained in the above section. 
    * Second, our aggregation weights are different from those in APPNP. In APPNP, the weight of the original information is $\alpha$ which is identical for all nodes. In GTCN, we use the diagonal entries of the normalized adjacency matrix as the aggregation weights of the original information for all nodes which is related to the degree of each node. In GTAN, we use attention based weights. Most GCN-like models are only different in their aggregation weights. For example,  the difference between the propagation rule in GIN[1] and GCN is just a hyperparameter $\epsilon^{k}$. If we do not acknowledge the interpretation behind such difference as contributions, there would be no need to publish those work.
     *Third, APPNP claims that  their propagation scheme "can effectively use even infinitely many neighborhood aggregation layers". We hold a different opinion towards this claim. The propagation rule in APPNP is $H^n = (1-\alpha)AH^{n-1} + \alpha Z$, where $H^0 = Z$. If we expand this expression to the original hidden feature $Z$, we have the following expression:
$$
H^n = (1-\alpha )^n A^n Z + (1-\alpha)^{n-1} \alpha A^{n-1} Z + (1-\alpha)^{n-2} \alpha A^{n-2} Z + \ldots + (1-\alpha) \alpha A Z + \alpha Z
$$
         * If $\alpha$ is small (The empirical tests in APPNP use $\alpha = 0.1$), the first term dominates in the above formula, making APPNP similar to simple GCN [2].
         * If $\alpha$ is large, the original information of each node dominates, which loses the neighborhood information. 
         * If $\alpha$ is not too large or small and is between (0,1), the high order terms would vanish quickly. Therefore the performance of APPNP should converge to some point when stacking multiple layers regardless of the data structure, and cannot go infinitely deep.
The high order terms in our GTCN model do not vanish.

Based on the above discussion, we can see that a hyperparameter makes big difference in model interpretation and model performance.

[1] Xu, K., Hu, W., Leskovec, J., & Jegelka, S. (2018). How powerful are graph neural networks?. arXiv preprint arXiv:1810.00826.

[2] Wu, F., Souza, A., Zhang, T., Fifty, C., Yu, T., & Weinberger, K. (2019, May). Simplifying graph convolutional networks. In International conference on machine learning (pp. 6861-6871). PMLR.

 We appreciate that you are open to change your score based on our improvements. The only point that we did not explain in your original feedback might be the time complexity of $O(k|E|)$. We rewrote that part to avoid any confusion. But we will explain this time complexity to you here. In our original manuscript, we mentioned that
> Although DAGNN achieves state-of-the-art performance, such multi-hop scheme mostly only adapts to GCNs from the model complexity perspective, as the sparse matrix multiplication trick can be applied to complete the computation for the k-hop aggregation $\tilde{A}^k(XW) = \tilde{A}(\tilde{A}^{k-1}XW)$, resulting a time complexity of $)(k|E|)$ which is comparable to that of the GCN.

We mentioned this time complexity not to criticize DAGNN, but to support that it is doable to compute $A^k(XW)$ by utilizing the sparsity of the adjacency matrix. However the limitation of DAGNN is that it may only work with simple aggregation function like the one in GCN. It is computational challenging to combine it with attention-based GNN models, because it requires computation of the attention weight for each pair of a node and its k-hop neighbor, which equals to the non-zero entries of $A^k$. $A^k$ could be a dense matrix despite that $A$ is usually sparse.

We now have removed the time complexity of computing $A^k (XW)$ to take away any confusion. We emphasize the limitation of DAGNN in the revision to make it more clear.

GCN or GAT is quite simple and scalable, so achieving the same order complexity as GCN is a good thing.

Other than this concern, could you be a bit more specific about your concerns that we haven't addressed yet please? We would like to try our best to address all your concerns and further improve our work. 

Thank you!

 We sincerely appreciate all reviewers' constructive feedback which helps us greatly improve our work. We have carefully revised our paper according to all comments. Here we summarize the major revision we made to our paper.

1. We abbreviate our Graph Tree Network as GTreeNet throughout the paper to make it more concise.

2. In the Introduction, we restate the necessity of using larger neighborhood information in graph learning with supported references. We also revised our discussion about DAGNN to make it more clear to readers about its limitation.

3. We rewrite Section 2 about our approach to make it more logic and organized. We first introduce our notations used throughout this paper. We then illustrate the tree representation of a sample graph. We add our assumption about message passing under the tree representation that a node $u$ at any k-hop preserves its initial information prior to receiving information from its child nodes. With this assumption, the message passing rule in GTreeNet is derived naturally. We add a figure in Appendix to illustrate the whole GTreeNet to help readers better understand our learning structure. We compare our general message passing scheme to the one used in GCN-like models to emphasize the difference. Next, we propose two graph tree learning models derived from the general message passing rule - GTCN and GTAN models. We formulate the specific message passing rule in GTCN and GTAN in node level to help readers understand the connection between our specific models and the general message passing rule in GTreeNet. In the section of each model, we make comparison with GCN, APPNP, Tree-LSTM, and GAT models to explain the differences between our models and other models, and explain why our models can go deep.

4. We add one more experiment on obgn-arxiv dataset as suggested by Reviewer 3AAc to further support our model performance.

5. We add Table 4 of averaging training time per epoch to show runtime of our models as requested by several reviewers.

6. We revise Section 3.3 to make our statement more rigorous. In the original paper, we claim that adding a non-linear function after each propagation layer degrades model performance. This is not accurate. In fact, it is that adding an MLP layer after each propagation layer degrades model performance. We compared the results by adding an MLP layer to each propagation layer to our GTCN and GTAN models, and test the model performance at different depths.

7. We revised our Conclusion as suggested by Reviewer T8A2. We removed the formulation for the heterogenous graph model, and summarize contributions of our work.

We welcome further discussions! Thank you!

 I thank the authors for their response, but the response does not address my concerns.

First, the authors do not answer: What's the novelty of the tree structure? 

The classical GCN can also be interpreted as a tree. For example, just see Jure's GCN tutorial. Then what's new for the tree proposed in this paper?

The authors keep saying that their formula is different from GCN. However, I'm asking about the reason. How does such a difference in formula result from tree structure? 

A good way to explain it is to put two figures. One is a classical GCN tree and another is a new tree. Then highlight different message flows in these two trees with different colors

Second, the difference between this paper's formula and APPNP is just a hyper-parameter... The technical contribution is almost none






The paper studies the task of network embedding. The authors propose a Graph Tree Networks (GTree), a deep graph neural network architecture that originates from the tree representation of the graphs. In particular, the proposed approach conduct message passing upward from the leaf nodes to the root node to update the root node’s hidden features. Experiment validates the effectiveness of the proposed model, ### Strength

- The idea of utilizing the tree representation is generally sound.


### Weakness

- From the second paragraph, the authors seem to make their tale by pointing out the complexity issue in the existing models (e.g. "time complexity of O(k|E|)", "could be a too large number" ). However, there is no interpretation on how their proposed model can address this issue. Particularly, the complexity of the proposed model is still in the same order as the vanilla GCN and GAT. And in the experiment, they only evaluate their proposed model in terms of effectiveness, which is hard to give a good end to the tale in the introduction.
- The approach section is pretty chaotic. I am really hard to follow the key idea of their proposed model, and the model formulation is not tidy enough. E.g. Equation (5) and (6) could be more concise.
- In the conclusion, the authors further propose the Heterogeneous Graph Tree Attention Network (HGTAN) with its formulas. But this proposed model hasn't been validated by any means, and should not be in the conclusion section. Instead, the conclusion section should focus more on the Graph Tree Network and the conclusion should connect to the experimental result, rather than some subjective statements, such as '"We believe xxx".
- The writing is a critical issue for this paper. There are too many grammatical issues throughout this paper. And many statements lack their justification. E.g.Section 2, the first sentence, "Generally graph learning based on direct neighbors only does not yield the best performance." needs a justification.
- I don't think message passing through tree structure nodes would be a general scheme for all the nodes, since some nodes may not have any hierarchical structure. The analysis on which types of nodes could benefit more from the proposed tree-based message passing scheme should be added to further validate the generalization ability of the proposed model.
- "self-interpretive " has not to be interpreted and justified in the paper. - The writing of the paper should be significantly improved.
- The motivation is not well stated in a good tale and the proposed model cannot address the issues stated in the motivation.
- Many statements are too subjective and are not be well justified.

 I thank the authors for their answers and their improvements. Some of my concerns have been resolved, although the changes are not well highlighted. I will reflect on these improvements to my score.

 Thank you very much for your acknowledgement on our core ideas and constructive feeback. We will elaborate on your each comment in below.
1. The average training time per epoch in ms/epoch on the four DGL datasets tested on RTX 3060 12GB are summarized in the following table:
| Models | Cora | Citeseer | PubMed | Coauthor-CS|
|--------------------|---------|-------------|------------|-------------------|
|GCN (hop=2)   |3.5      |3.3           |3.6         |      7.4           |
|GAT (hop=2)    |4.7      |4.6           |5.0         |      12.0         |
|GCN (hop=10)  |14.3      |13.9         |19.1         |      25.9        |
|GAT (hop=10)   |17.6      |17.8           |21.8         |      32.7         |
|APPNP (hop =10)      |3.7      |3.5         |3.8         |      12.0         |
|DAGNN (hop=10)      |4.7      |4.6           |5.1         |      13.2      |
|GTAN (hop=10)      |17.7    |18.1           |19.4         |      34.4        |
|GTCN (hop=10)     |3.7      |3.6           |8.1        |      18.6        |

2. We add the tests on the ogbn-arxiv dataset to our revised manuscript. Our GTAN model outperforms APPNP and DAGNN. We found that many datasets in ogb are heterogenous graph that we are very interested in. We are currently working on extending our models to heterogenous graphs and have obtained encouraging results. We would like to explore more datasets in ogb to enrich our work in the future. Thanks for your suggestion!

3. We have carefully revised our description. To be more accurate, the model performance degrades when an MLP transformation layer is added after each propagation. An MLP transformation introduces additional trainable parameters which may cause worse fitting especially when training samples are limited. This performance degradation will be more significant when stacking multiple propagation layers, as the number of additional parameters accumulates quickly when models are going deep. In the original paper of DAGNN, the authors claim that the intertwined propagation and transformation is the key reason that causes performance degradation when stacking multiple propagation layers.

We hope our revision and explanation address your concerns. Thanks!

 Thank you for acknowledging our core concepts. We appreciate your feedback which helps us improve our paper. We will elaborate on each of your comment in below.

1. The major contribution of our work lies in two parts: 1) We formulate a general messing passing scheme by rethinking the graph as tree which covers APPNP and Tree-LSTM. This message passing scheme is more intuition and interpretive with the assumption in our tree representation. This may open new research opportunities by exploring different combination of aggregation function and transformation function. 2) We propose two graph tree learning models - GTCN and GTAN which can go deep by stacking multiple propagation layers without requirement for finely tuned hyperparameter as that in APPNP. The path information is self-contained when stacking propagation layers which may allow our models to be extended to work on heterogenous graphs.

2. We show the tree representation in Figure 1 just for illustration purpose to help readers better understand the concept behind our models. In practice, there is no need to transform from graph to tree structure. The topology of neighborhood is readily obtained from the adjacency matrix.

3. We respectfully disagree with your claim of unclear contribution because the method is intuitive and straightforward. We think it is a merit if a method is simple and intuitive yet have superior performance. We do agree that we should emphasize our contributions to make them more clear to readers.

We hope our revision and explanation would change your mind toward our paper. Thank you!

 We sincerely appreciate your detailed and constructive feedback which helps us greatly improved our paper. We have carefully revised our paper according to your each comment. We will elaborate on each of them in the following discussion.

1. There might be some confusion in our description. We have carefully revised our description in the Introduction to make it more clear. The DAGNN-like models compute the k-hop aggregation directly in one step. Such multi-hop aggregation scheme only work with simple aggregation function as the one used in GCN. It could be computationally challenging to combine such multi-hop aggregation scheme with attention based GNN models. When working with attention based GNN models, such aggregation scheme requires computation of the attention weight for each pair of a node and its k-hop neighbor. The number of attention weights in the k-hop aggregation equals to the total number of non-zero entries in the k-th power of the graph adjacency matrix $A$. Although the adjacency matrix is usually sparse, its k-th power could be non-sparse especially for undirected graphs. For example, in the Cora dataset, $A^5$ has ~2.2 million ono-zero entries while $A$ has only ~10k non-zero entries. Our models achieve large receptive fields by stacking multiple propagation layers which only requires aggregation from direct neighbors in each layer. The number of attention weights required in our GTAN model is $L|E|$, where $L$ is the depth of graph tree, |E| is the number of edges.

The average training time per epoch in ms/epoch on the four DGL datasets tested on RTX 3060 12GB are summarized in the following table:

| Models | Cora | Citeseer | PubMed | Coauthor-CS|
|--------------------|---------|-------------|------------|-------------------|
|GCN (hop=2)   |3.5      |3.3           |3.6         |      7.4           |
|GAT (hop=2)    |4.7      |4.6           |5.0         |      12.0         |
|GCN (hop=10)  |14.3      |13.9         |19.1         |      25.9        |
|GAT (hop=10)   |17.6      |17.8           |21.8         |      32.7         |
|APPNP (hop =10)      |3.7      |3.5         |3.8         |      12.0         |
|DAGNN (hop=10)      |4.7      |4.6           |5.1         |      13.2      |
|GTAN (hop=10)      |17.7    |18.1           |19.4         |      34.4        |
|GTCN (hop=10)     |3.7      |3.6           |8.1        |      18.6        |

2. Thanks for your suggestion. We have carefully revised Section 2 to make the presentation of our approach more concise and organized, and hope it conveys our core concept to readers smoothly and logically.

We first introduced our notations used throughout this paper.

We then present the tree representation - the core concept behind our Graph Tree Network (GTreeNet). We emphasize the assumption in our tree representation that a node $u$ at any k-hop preserves its initial information prior to receiving information from its child nodes. This leads to the general message passing rule in our GTreeNet. We add a figure to illustrate the whole GTreeNet to make the concept more clear to readers.

We then explain the difference between the message passing rule in our GTreeNet and the general message passing rule adopted by classic GCN-like models.

Next, we propose our GTCN and GTAN models. We first formulate the message passing rule in node-level to be consistent with the general message passing rule in GTreeNet, and explain how to derive our model from the GTreeNet by using specific combination of  aggregation function and transformation function. 

We also make the equation more tidy according to your suggestion. Please refer to our latest version for the details.

3. Your suggestions make great sense. The reason that we put the formulation of our heterogenous model in conclusion was to establish authorship as our preliminary results were very promising that our paper will come out soon. In the latest revision, we have removed the formulation of heterogeneous model.

4. Thanks for your advice. We have carefully revised our manuscript and had it proofread by a reputational scholar. We have carefully check our statements and claims, and added proper references to justify our statements.

5. We have added a figure to illustrate the whole GTreeNet to help readers better understanding how the graph tree works. For an isolated node, its features are updated by its initial hidden features. That is, $h_u^k = z_u$ for all $k$.

6. We added the assumption behind our tree representation of graphs in revised Section 2.1. We assume a node $u$ at any k-hop preserves its initial information prior to receiving information from its child nodes. With this assumption, a node's features are naturally updated by aggregating its child nodes and its own initial features in the tree representation, as messages propagate upward from leaf nodes to the root nodes in the tree.

We hope our revision and explanations address your concerns and change your mind toward our work. Thanks!

 We appreciate your time reviewing our paper and feedback. However, we respectfully disagree with your conclusion.

We have carefully revised our paper to emphasize the difference of the messing passing scheme in our GTreeNet and classic GCN-like models. 

The general message rule in our GTreeNet is shown in Equation (1) in Seciton 2.1 as below:
$$
h_u^k = f_k \left(\text{aggregate} \left(\phi(x_u), \\{h_v^{k+1}, \forall v \in N_u \\} \right) \right)
$$
where $k = L-1, L-2, \ldots, 0$. $L$ is the tree depth, i.e. the maximum hop of neighbors under consideration. $k$ is the number of hop away from the root node.

The general message rule in classic GCN-like models is shown in Equation (2) in Section 2.1.1 as below:
$$
h_u^n = f_{n-1}\left(\text{aggregate} \left(h_u^{n-1}, \\{h_v^{n-1}, \forall v \in N_u \\} \right) \right)
$$
where $n$ is the number of propagation layer.

The major difference between the above two propagation rules is that we update a node's hidden features by aggregating its initial hidden features with its neighbors' features, while the GCN-like models update a node's hidden features by aggregating its hidden features and its neighbors' hidden features from the last propagation layer ($\phi(x_u)$ v.s. $h_u^{n-1}$). Our propagation rule preserves the local information when stacking multiple propagation layers, while GCN-like models do not.

The assumption behind our tree representation is that a node $u$ at any k-hop preserves its initial information prior to receiving information from its child nodes. With this assumption, using $\phi(x_u)$ in aggregation in more interpretive than using $h_u^{n-1}$. 

Next we will compare our GTCN model with APPNP.

For easier comparison, we will continue use node-level propagation rule for comparison.

The propagation rule of our GTCN model is shown in Equation (3) in Section 2.2 as below:
$$
h_u^k = \sum_{v\in N_u} A_{uv} h_v^{k+1} + A_{uu} z_u
$$

The propagation rule of APPNP is shown in Equation (6) in Section 2.2.1 as below:
$$
h_u^n = \left(1-\alpha\right) \left( \sum_{v\in N_u} A_{uv} h_v^{n-1} + A_{uu} h_u^{n-1} \right) + \alpha z_u
$$

APPNP uses a node's hidden features from the last propagation layer in the aggregation while our GTCN model does not. Although APPNP is proposed based on the personalized PageRank, it can be derived from our GTreeNet by adding self-loops in the graph. In the tree representation of the modified graph, node $u$ is also a child node of itself. In this case, the general propagation rule in the resulting tree representation becomes:
$$
h_u^k = f_k \left(\text{aggregate} \left(\phi(x_u), h_u^{k+1}, \\{h_v^{k+1}, \forall v \in N_u \\} \right) \right)
$$

Although APPNP uses a node's initial features in the aggregation, the concept of personalized PageRank behind it makes it essentially different from our GTCN model. APPNP requires a fining tuned hyperparameter of $\alpha$ to adjust the influence of neighborhood. Our GTCN model does not require such fine tuned hyperparameter.

We hope our explanation may change your mind towards our paper. 




This paper proposes a message-passing formula derived from the tree representation of a graph.
However, the proposed method has no innovation. I see no difference between the classical message passing scheme and the tree structure proposed in this paper.
The classical message passing scheme can also be described by a tree where each node's neighbors are its children.

The specific formula proposed by the paper is almost the same as APPNP [1]

[1] Predict then Propagate: Graph Neural Networks meet Personalized PageRank. ICLR 2019 This paper has no novelty.

The authors propose a novel neighborhood aggregation technique for GNNs based on the tree representation of the graph. Their proposed methods GTCN and GTAN pose comparable time complexity to that of GCN but empirically yield better results on various popular benchmarks.  Strenght: 
1. The model architecture allows the network to encode path information crisply.
2. The Graph tree network is more expressive than APPNP.


Weakness
1. I find the empirical evaluations are somewhat limiting. Questions/ Clarification/ Suggestions:
1.  I understand that the time complexity is comparable with GCN, however, I am still curious to know about training time (per epoch) for al the proposed models. 
2. Open Graph benchmark (https://ogb.stanford.edu/docs/leader_nodeprop/) provides a wide variety of tasks and associated datasets that are more interesting than these benchmarks. I would encourage the authors to apply their model to some of them.
3. Why does the performance degrades significantly when a nonlinear layer after each propagation layer is added? The authors have never addressed that in the paper. I think pressing a bit more along that line might lead to a better interpretation of the message passing of the model.

I would be happy to increase my score if some of the above-mentioned concerns are properly addressed.

In the paper, a self-interpretive graph learning architecture is presented based on the tree representation of graphs.The message passing scheme is natural and intuitive. Two models, Graph Tree Convolution and Graph Tree Attention Network, are given with SOTA performance.  This paper provide two deep learning models to dealing with the tree representation of the graphs. 

Strength: 1. The message propagation is clear and explanable, which is dfifferent from conditional vanilla CCN， GAT and other dervatives.  2. The method is easy to follow and intuitive. 3. The performance reaches that of SOTA.

Weakness:  1. The proposed method is intuitive and straightforward. So the contribution of the paper is unclear.  This paper provides a simple and intuitive way to carry out graph learning based on the tree represnetaion of graph. And the performance of the presented models looks convincing. Source code is give for evluation. But I still have the following concerns:
1. What are the major contributions of the paper? 
2. The time complexxity should include the transfering time from graph to tree structure? Especially for the large scele graph data? 

Meta-review: While I understand and have empathy with the authors' viewpoint of their work and novelty, this unfortunately has not reached reviewers' hearts in the way that they intended. There has been no strong support for acceptance, as the questions about the amount of novelty piled up. Some interactions between authors and reviewers happened. It is clear that the authors made an effort to show the differences with respect to other tree-related models and algorithms, as well as to highlight the strengths of the approach which was claimed to be too similar and simplistic. I believe the interactions helped with improving the first viewpoint of reviewers, however this improvement has not been enough, as reviewers did not significantly changed their stances. This is a short process and indeed it is not easy to change first impressions. If anything to add is that I hope that the impressions can be used to give a new presentation to the work that will enhance the work and its view by others.

Decision: Reject