Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: **Summary**
The paper proposes a GNN model by incorporating gradient boosting. In the proposed BGNN, the input feature on the graph is learned by the gradient boosting model. The processed feature then becomes a new feature for a GNN model following the gradient boosting. Several experiments demonstrate the improvement of the performance for a tabular feature and graph-structured datasets. The running time of Res-GNN/BGNN is shown to have a significant reduction as compared to the plain GNN methods.

**Comments and questions**
1.  The paper proposed a simple model combining Boosting and GNN methods, which can effectively learn the heterogeneous features of graph-structured data.
2.  The performance of BGNN, including the training speed and regression error, is good as compared to the GNNs for datasets with tabular features.
3.  What is the performance of the proposed model on other node property prediction tasks such as Open Graph Benchmark?
4. The BGNN has two stages: a gradient boosting for graph features and graph neural networks for graph-structured data, including features plus graph structure. So, the computational time should be the sum of time of GBDT and GNN. Why is the time becoming smaller than the plain GNN for the same datasets?
5. How many layers do the BGNN exactly use for experiments?
6. Will the GBDT apply to link prediction tasks? 

review_title: Review for "Boost then Convolve: Gradient Boosting Meets Graph Neural Networks"

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper aims to learn from graphs with tabular node features. Existing methods are only designed to handle either tabular data, such as gradient boosting decision tree (GBDT), or graph-structured data, such as graph neural networks (GNNs). This paper naturally extends GBDT to deal with graph-structured data and train it together with GNN in end-to-end fashion.

This paper is clearly written and easy to follow. I enjoyed reading this paper. The idea is clear and well-motivated. The way of combining GBDT and GNN looks natural and reasonable. The essence of GBDT is to approximate the gradient of the undifferentiable part of the model by selecting a weak learner h from H (eq. 2). This proposed training method seems valid and effective on semi-supervised regression.

My main concern is the proposed method is not working well on semi-supervised classification tasks. It performs worse than CatBoost (pure GBDT method without using the graph) on the two datasets Slap and DBLP for semi-supervised node classification, which is probably why the authors put the experiments in the Appendix. 

Since node classification is a benchmark task and, in my opinion, more common and important than semi-supervised regression, I have some doubts of the practicality of the proposed method. 

============Post Rebuttal=============================

Thanks for the additional experiments and the updates. The new results are informative. 




review_title: A simple method combining GBDT and GNNs to deal with graph datasets with tabular heterogeneous features

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: This paper presents a new architecture called BGNN (Boost-GNN), which combines the benefits of GNNs (Graph Neural Nets) with GBDTs (Gradient Boosted Decision Trees).

Basic idea:
* GBDTs work well with *heterogeneous* tabular data.
* GNNs work well on *graphs* with *homogeneous* sparse features.
* BGNNs work well on *graphs* where the nodes contain *heterogeneous* tabular data.
* BGNNs is optimized end-to-end and seems to obtain great SOTA results!

An example of the data BGNN works well on is in social networks. E.g. each node could be a person with heterogeneous characteristics such as age, gender, graduation date.

The main trick is in how to train this end-to-end effectively. This is done by iteratively adding trees that fit the GNN gradient updates, allowing the GNN to *backpropagate* into the GBDT.

More in detail:
* GBDTs make a prediction for each node in the graph.
* The GNN reads the output of the GBDTs and corrects the predictions based on the graph structure.
* When doing gradient descent on the GNN the authors critically also optimize with respect to the input features. The difference between the optimized input features and the original input features becomes the new objective for next round of update on the decision trees.

Results seem very strong across all tasks considered in the paper.

Strengths:
* The paper presents an interesting and convincing analysis of why GBDTs are so advantageous for tabular data, while GNNs are the best choice for graph data.
* The combination of GNNs and GBDTs is not trivial. Several papers are cited attempting the combination of neural nets and gradient boosting, but they are reported to be computationally more expensive and not as powerful.
* The detail about computing updated targets for GBDTs while doing gradient descent on the GNN seems particularly interesting and creative.
* The results obtained by this paper are impressive, surpassing SOTA models by considerable margins.
* The paper also finds that representations learned with BGNN have more discernible structure, suggesting that they are more interpretable.

review_title: A very creative method for training combinations GNNs and Gradient Boosted Decision Trees end-to-end with strong results!

final_score: rating: 5: Marginally below acceptance threshold
review_content: Review: This paper proposes a fusion of GBDT and graph neural network that works on graphs with heterogeneous tabular features. Previous approaches are computationally heavy and do not consider graph-structured data and suffer from lack of relational bias imposed in GNNs. The proposed method is a new ensemble tree method which alternates between functional gradient step in GBDT (which train on the current latent features) and SGD training of graph neural network (to generate the latent features which are fed into the subsequent trees).

Reasons for score:

Overall, I am leaning toward rejecting this paper. Overall, the paper is an empirically motivated paper focusing on the construction of pipelines which depend on already established methods, GBDT and GNN. Having advantages of both methods is nice, but it is hard to glean why this proposed method works well on theoretical basis.

+Positives:
- The proposed end-to-end training combining GBDT and GNN is easy to implement and clearly described.

- improving GBDT to work on graph-structured data is a novel idea.

- Comparison in the experimental section considers a handful of state-of-the-art approaches.

Concerns:
- Experimental results are mainly concerned with regression tasks. Having some results on classification would be nice.

- The contribution of the paper is mainly empirical and offers little intuition on why the proposed method improves upon GNN and GBDT.

review_title: A novel gradient boosted decision tree model that works better on graph representation data

Meta-review: The paper proposes an interesting architecture that dues Graph Neural Networks (GNN) and Gradient Boosting Decision Tree. This new architecture works on graphs with heterogeneous tabular features and BGNNs work well on graphs where the nodes contain heterogeneous tabular data and is optimized end-to-end and seems to obtain great SOTA results. End to end learning is done by iteratively adding trees that fit the GNN gradient updates, allowing the GNN to backpropagate into the GBDT. All reviewers agreed that the idea is interesting, the paper is well-written, and the results found in the paper are impressive. In addition, author response satisfactorily addressed most of the points raised by the reviewers, and most of them increased their original score. Therefore, I recommend acceptance.
Decision: Accept (Poster)