Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors propose a new method for integrating graph-based models with boosting. This is done using the typical method involving residuals and weak-learners, but adding a step where information is propagated in the graph. The approach is also simple, as no GNNs or other auxiliary models are required. It is also shown how the meta-loss introduced by the authors provides convergence given some moderate assumptions. According to the experiments reported, the proposed model is better than the current state of the art in the considered domain.
main_review: The notation used is easy to understand, as is the mathematical explanation in section 3, which is presented in a comprehensive but concise manner.
"for EBBS we must fit the weak-learners to gradients from both the training and test nodes". This is the sentence that I am most concerned about, as the use of test data in the training phase may render the results obtained invalid. Although the authors give their own explanation of why the test nodes should also be used during training, i.e. for the propagation of information in the graph, if the test labels are used during the train there is no longer any separation between train and test. I have this doubt because the labels are used to calculate function-space gradients. Is this correct?

Algorithm 1 could be described in a little more detail.

The analysis of the convergence of the method by theorem is very good. The remarks connected to the theorem are interesting, but could have been treated in more detail (they are in part in the supplements).
If experiments are done with different random seeds (as stated), then results in Table 1 should be reported with their corresponding standard deviation or confidence interval.
Why are some results reported with different decimal places in the table? I am talking specifically about the CS column, but also for the Slap, DBLP and Phy columns one decimal place could be added.
If the datasets were taken from Ivanov & Prokhorenkova (2021), why was Wiki not taken? The reason should be that, being a homogeneous dataset, then, as explained by Ivanov & Prokhorenkova (2021), "neural network approaches are sufficient to achieve the best results". It would still be interesting as a comparison. Also as a comparison with them, the House and VK datasets could also be used for classification. They also report the standard deviation of all results.
Also, the results in the table match those of Ivanov & Prokhorenkova (2021), but I do not understand why their LightGBM results row has become the CatBoost row in this article for the Slap, DBLP and OGB-ArXiv datasets. Is this perhaps an error?
Although the difference between OGB-ArXiv and the other datasets is properly explained, I think it still makes sense to put those results together with the others in Table 1.
"Although tabular graph data for node classification is widely-available in industry, unfortunately there is currently little publicly-available, real-world data that can be used for benchmarking."
This sentence is very vague and I am not fully convinced of its veracity.

The mention of the method called CatBoost+ is interesting, but it is given too little space. Why is it not considered in "ours"? If the idea is picked up by some other work, let it be mentioned properly.
"revealing that it may be more robust to non-ideal use cases". That's why it might be interesting to add homogenous datasets and see if it applies there too.
While in the main part it says "This suggests that in new application domains it may conceivably be easier to adapt EBBS models", in the supplements it says "It shows EBBS can be run with mostly shared hyperparameters across all datasets". I don't think there are enough experiments/results to say that, but I'd stick with "suggest" in the supplementary materials as well. Maybe add a sentence about the possibility of exploring this area more in future work.

After the rebuttal I have strengthen my opinion on the quality of the paper. I believe it is a nice contribution to the field.
summary_of_the_review: The work is well structured, with a good theoretical basis to support the proposed methodology. The empirical results are very promising, although the small amount of datasets combined with the lack of confidence intervals does not allow for meaningful conclusions to be drawn.
The only major doubt concerns the use of test data in the training phase, which may have compromised the whole experiment.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Following in the success of boosting methods for tabular data, this paper introduces a new boosting approach for data that graph-based with tabular features. The proposed approach, efficient bilevel boosted smoothing (EBBS), has convergence guarantees as well as empirical successes compared to competing methods. 
main_review: **Summary** This paper investigates tabular, graph-based data for classification and regression tasks. The proposed approach is an end-to-end, bilevel, combination of label propagation and boosting. The authors contribute not only an empirical analysis of the proposed approach on 8 datasets demonstrating the effectiveness of the proposed approach as well as a theoretical analysis. 

**Merits** I believe that this is a strong paper that clearly outlines a proposed approach for boosting in this non-iid setting of graph data. The proposed approach has a convergence guarantee and is shown to be very effective empirically. Overall, this seems to be a strong result. The supplemental material seems to give statistical significance of the improvements shown. Compared to the best previous method BGNN, combining boosting and GNNs, EBBS achieves stronger empirical results. The algorithms and theoretical results are discussed as well. 

**Weaknesses** Here are a few concerns / suggestions:
* It could perhaps be made stronger by including some of the additional analysis that is in the supplemental material that investigates the trade-offs and ablations of the approaches, in the main body of the text. 
* I think that the paper could be made much stronger with a simple motivating (perhaps synthetic) example that illustrates where and when EBBS can be useful compared to competing methods. While convergence guarantees and motivations are described, a clear simple example (which might further be useful in using ablations to identify contributions of different parts of the solution) could strength the paper. 

**Minor Notes**
* Why is "GraphData" one word in the title?
* Figure 1 would be easier to read if Y-axis was the same in both plots
summary_of_the_review: This paper provides both theoretical and empirical results for a boosting method for graph structured data. The results appear to advance the state of the art and the submission seems to have valuable contributions.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new way to integrate graph-based models with boosting based on principled meta loss, named EBBS. In experiments, the proposed method outperforms tabular baselines, GNN baselines, and some hybrid strategies like BGNN over some node classification/regression datasets.
main_review: Strengths:  This paper proposes EBBS, efficient bilevel boosted smoothing, a novel way to combine GNN and Gradient boosting for learning tabular graph data. The addressed problem of integrating boosting into GNNs is very interesting to me. Also learning over tabular graph data should receive a wide audience given its importance in the industry. Empirical experiments show EBBS outperforms baseline methods on multiple node classification and node regression datasets. 

Weakness: In my opinion, the main weaknesses are in writing/presentation and reproducibility.
First, I feel the writing of Section 3 can be improved to avoid readers' confusion. For example
  - We can be more clear about how Eq 2 is rooted in Zhou et al 2004. In fact, I didn't get it when I checked the referenced paper. 
  - In (2), are both Z and \theta learnable? 
  - P is binded twice, once in P* (Eq 3) and once in P^{k} (Eq 6)
  - In Eq 7, what is for inner level optimization and what is for outer level optimization?
  - Is "Graph-Aware Propagation Layers" terminology used in the literature?
Second, it seems that the proposed method EBBS will be incorporating test nodes during training. Will this cause test information to leak into the training process? Is there any specific preprocessing to avoid leaking? Is EBBS easy to implement?


Minor issues (typos, formats): 
- "on top model leaderboards" -> on the top of model leaderboards
- the template seems a bit different from the normal one, especially the font and the colour of the citation text
- "whereby the end-to-end training of a bilevel loss is such that values of the boosted base model f ebb and flow across the input graph producing a smoothed predictor f" -> not sure if there is a grammar issue 
- "use mi and to reference" -> delete "and" 
summary_of_the_review: Overall, I like the problem the paper aims to address - how to better combine GNN with boosting methods for learning on tabular data. The paper proposes a novel way to address this problem, which is based on a principled meta-loss. Empirical results show the effectiveness of the results. I feel the paper can be improved more by iterating on the formulations in Sec 3.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors present a new approach to combine the boosted decision tree classifiers with a graph propagation model, which is important in handling table input data. The approach casts the graph propagation as an optimization problem, where the input node features are generated by boosted decision trees. The gradient can be taken in the functional space to learn the decision trees to minimize a unified loss. The final algorithm is shown to minimize the unified loss in a principled manner. The superior performance is demonstrated over the existing BGNN model.

main_review: Strength
- The approach nicely defines a single objective that the model (graph propagation + decision trees) optimizes.
- Empirically, there is a nice improvement over the existing BGNN.

Weakness:
- The studied problem does not seem particularly novel to me, especially given BGNN. Given BGNN, the scope seems a bit narrow to me (although I acknowledge that the authors solve the problem in a potentially better way than the BGNN paper).

Comments/questions:
- I am curious to see the result of XGBoost + C&S (e.g., use XGBoost as the base predictor in C&S).
- Does the framework supports any propagation rules beyond (6)? I would be curious to see how general the method is.

summary_of_the_review: Overall, the approach seems sound and principled, although the scope is a bit narrow. Hence, I will give the weak accept. I would also like the authors to address my comments/questions.
review_title: null

Meta-review: The paper addresses a problem encountered in many real-world applications, i.e. the treatment of tabular data, composed of heterogeneous feature types, where samples are not i.i.d. In this case, learning is more effective if the typically successful approach for i.i.d. data (boosted decision trees + committee techniques) is combined with GNN to take into account the dependencies between samples. The main contribution of the paper with respect to previous work in the field is the introduction of a principled approach to pursue such integration. One important component of the proposed approach is played by the definition of a specific bi-level loss (efficient bilevel boosted smoothing) that allows for convergence guarantees under mild assumptions. Both theoretical and experimental contributions are sound and convincing, justifying the claimed merits of the proposed approach. Another strong point is the fact that the proposed approach is general and amenable to support a broad family of propagation rules. One weakness with the original submission was presentation, mainly because some key information was confined into the supplementary material. The revised version addressed this problem and added some more empirical results that confirmed the superiority of the proposed approach.
Finally, the fact that learning over tabular graph data is very important in Industry, the proposed approach may be of interest for a wide audience.
Decision: Accept (Spotlight)