TabGraphs: A Benchmark and Strong Baselines for Learning on Graphs with Tabular Node Features

Authors: Gleb Bazhenov, Oleg Platonov, Liudmila Prokhorenkova
Primary area: datasets and benchmarks
Keywords: graph machine learning, tabular machine learning, graph neural network, gradient boosting, benchmark

Abstract:
Tabular machine learning is an important field for industry and science. In this field, table rows are typically treated as independent data samples, but additional information about the relations between these samples is sometimes available and can be used to improve predictive performance. Such information can be naturally modeled with a graph, hence tabular machine learning may benefit from graph machine learning methods. However, graph machine learning models are typically evaluated on datasets with homogeneous, most often text-based node features, which are very different from heterogeneous mixtures of numerical and categorical features present in tabular datasets. Thus, there is a critical difference between the data used in tabular and graph machine learning studies, which does not allow one to understand how successfully graph models can be transferred to tabular data. To bridge this gap, we propose a new benchmark of diverse graphs with heterogeneous tabular node features and realistic prediction tasks. We use this benchmark to evaluate a vast set of models, including simple methods previously overlooked in the literature. Our experiments show that graph neural networks indeed can often bring gains in predictive performance for tabular data, but standard tabular models can also be adapted to work with graph data by using simple graph-based feature augmentation, which sometimes enables them to compete with and even outperform graph neural models. Based on our empirical study, we provide insights for researchers and practitioners in both tabular and graph machine learning fields.

Decision: reject
Reviewer scores: 3, 3, 5, 5

Reviews and discussion:
<thread>
creation date:
1729827693231

thread_type:
official_review

<thread_object>
creator: reviewer_xbq2

<content>
summary:
This paper proposes a benchmark for graph learning on single-table tasks(1 type of node and 1 type of edge), with heterogenous features. 
It constructs 11 tasks from 8 different datasets and ran experiments with GBDTs as well as GNNs on the tasks. 
The results show that GNNs outperforms GBDTs on the tasks authors proposed.

strengths:
Overall the writing is good and the presentation is clear. 
The experiments the authors provide are extensive, including different types of models, tabular dl models, GBDTs, GNNs, etc.

weaknesses:
The novel contribution is limited in this paper.

On page 2, the authors claimed that GNNs are typically evaluated on homogeneous features, but tabular learning typically has heterogenous features. I think they omit the work of PyTorch Frame (Hu, Weihua, et al. "PyTorch Frame: A Modular Framework for Multi-Modal Tabular Learning." arXiv preprint arXiv:2404.00776 (2024). ), which supports encoding heterogenous tabular data through deep encoders and can be used with PyG on downstream GNN tasks. It is good to include comparison to PyTorch Frame + PyG in your paper as it also fits into graph + heterogenous tabular data. 

The authors claim on page 4. "In contrast, our work is focused on single-table data with a single type of relationships
between entities in the same table.". However, it seems to me all the graph structure constructed by the authors can be represented by the graph structure in Relbench as well.

I also think the task type is too limited. Currently it only supports node classification and node regression. The authors did not mention link prediction tasks. Link prediction is a common tasks solved by GNNs and is offered by Relbench. It is unfair to say node level task is "the most common setting in modern graph machine learning".

questions:
In the experiment section, the authors only compared the performance scores. I think it's also good to compare the runtime of the methods, as GBDTs can be much slower on larger datasets as compared to GNNs and tabular models.

The authors propose two types of tasks, node regression and classification. I'd like to see some other task types being covered, for example, multi-label classification, or link prediction. For example, in the hnm dataset, can you add a task to predict new items that will be co-purchases?

The authors should make proper comparisons to existing relational learning benchmarks RelBench and 4DBInfer.

Happy to raise the score if the authors can address all the three comments above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730606296643

thread_type:
official_review

<thread_object>
creator: reviewer_czc7

<content>
summary:
The paper explores enhancing predictive accuracy in tabular machine learning by incorporating graph structures, proposing benchmarks to bridge the gap between heterogeneous tabular and homogeneous graph data features. Results indicate that graph neural networks boost the accuracy of tabular predictions, and simple graph-based enhancements can enable traditional models to effectively compete. Valuable insights are provided for both tabular and graph machine learning practitioners.

strengths:
- **Research Topic.** The studied problem is interesting and practically important for tabular machine learning.

- **Experiments.** Experiments are carried out on 10 datasets with 18 baseline methods, which is quite solid. And there are also enough discussion.

weaknesses:
- **Lack of Novelty.** The concept of constructing graph datasets from tabular data is not novel and has been previously suggested in multiple studies.

- **Writing.** The initial paragraph in the 'Machine Learning for Graphs' section of the related works is excessively lengthy, potentially hindering readability. Besides, The main text should provide more details about how tabular data is represented as graph data. The absence of these details in the main text may lead to confusion among readers.

questions:
- More discussion is needed on the differences between this paper and the previous paper on this topic.

- Why there are models that suffer from performance drop after utilizing PLR?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730686250154

thread_type:
official_review

<thread_object>
creator: reviewer_sjtv

<content>
summary:
The paper introduces a benchmark to evaluate graph ML methods on tabular data with heterogeneous features. The paper hopes to bridge the gap between tabular and graph machine learning studies. It finds that while GNNs can enhance predictive performance, simple adaptations of standard tabular models could also work well.

strengths:
- The experimental results are comprehensive. I appreciate the efforts from the authors to conduct experiments over 10+ baselines and 10+ datasets
- The paper cited a large number of related works, which helped the readers to know more about the field.

weaknesses:
- The problem setting is flawed. I do not agree that graphs with tabular node features are different from relational databases and can be redeemed as a new type of ML problem to work with. Indeed, the authors mentioned that "In contrast, our work is focused on single-table data with a single type of relationships between entities in the same table", and led the readers to Appendix C. But even after carefully reading the Appendix C, the paper still does not explain why the setting is different from a relational database with 2 tables (1 for the original table, the other for the additional relational information within the table). In that case, my understanding is that this paper is only a benchmark for relational databases with 2 tables, and only serves as a special case from the existing relational database benchmarks.
- Moreover, the authors did mention "Very recently, two benchmarks of large-scale relational databases have been proposed: RelBench (Fey et al., 2023) and 4DBInfer (Wang et al., 2024)." in the appendix. Given the high correlation with the proposed domain, they should be highlighted in the main paper and be thoroughly discussed, instead of hiding it within 1 sentence in the appendix. I'm not sure able the reason why the relevant discussions should not be mentioned in the main paper.
- The writing is a bit verbose. For example, the motivation in the introduction is weak. The first 2 paragraphs only introduce why we need tabular learning and graph learning, which is not relevant to the research problem in this paper.
- The paper makes little to none algorithmic contributions. Overall, I think this paper has not met the publication bar for ICLR.

questions:
- Why does this paper not emphasize the high relevance of ML for relational databases? Why should relevant discussions go into the Appendix? The "Machine learning for graphs" paragraph in the related work took almost a page, but it is less relevant to ML for relational databases for this topic since the setting is essentially just a relational database with 2 tables.
- Why working with homogenous graphs a benefit, rather than a limitation? Extending homogeneous GNNs to heterogeneous GNNs only takes a few lines of code in PyG, and it makes the GNN model more expressive. I don't find homogeneous GNNs present benefits with heterogeneous GNNs.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731905630665

thread_type:
official_review

<thread_object>
creator: reviewer_hwyk

<content>
summary:
In this work, the authors aim to study the usefulness of introducing graphs while working with ML for tabular data. To bridge the gap between graph based ML (E.g. techniques such as GNNs, Deepwalk, etc.) and tabular ML methods (e.g. GBDT) - the authors propose to create a benchmark of graph datasets with heterogenous tabular node features across domains such as fraud detection, road networks, etc - with varying number of nodes, edges and other commonly studied graph based statistics. The authors then use the created benchmark to evaluate different ML methods and their combinations.

strengths:
1. The authors propose multiple datasets which combine graph structure with tabular data -- specifically with heterogenous node data.
2. The authors compare a wide range of methods on the constructed benchmarks

weaknesses:
1. The authors appear to ignore the graph based tabular datasets and tasks given presented concisely in the survey --  https://arxiv.org/abs/2401.02143 which survey prior works which have employed GNNs for tabular data.
2. While the authors state "If it is possible to define meaningful relations between data samples, it is worth trying to convert the given data to a graph and experiment with ML methods that are capable of processing graph information, as it can lead to significant performance gains" - and given its largely a benchmark paper - they do not propose a standardized mechanism or methodology to augment any table with graphs - largely limiting the novelty of the contributions. 
4. The analysis accompanying the results presented on the benchmarks also feel handwavy - "it is important to experiment
with different design choices" - seems to suggest - it is prudent for the downstream users to try every possible method and architecture, and therefore not providing a way to prune the search space of methods or architectures for a given dataset/ problem
5. As the paper notes, the graph methods proposed here are not novel and have been employed by works such as https://arxiv.org/abs/2004.05718 and others -- even for heterogenous node data - and therefore the paper lacks novelty from the method perspective as well.

questions:
Please address the weaknesses section as applicable
</content>
</thread_object>

</thread>
<thread>
creation date:
1732207232941

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response

comment:
We thank all the reviewers for their feedback!

Some reviewers asked us to clarify the differences between the setting considered in our work and the setting of ML for relational databases (RDBs) and thus between our benchmark and RDB benchmarks such as RelBench and 4DBInfer. Below we explain these differences in detail.

There are two main axes of difference. The first concerns where the data comes from. In our work, we assume that data comes from a single table but there is additional information available about relations between the rows of this table. Note that single-table data is the standard setting of tabular ML, and such data appears in a vast range of real-world industrial and scientific applications. In contrast, the setting of ML for RDBs assumes that data comes from a relational database, i.e., a collection of tables with additional relationships between entries of different tables. It is true that, as reviewers notice, our single-table data can be represented as an RDB with a single table, and thus can formally be considered a special case of RDB data. However, it is a very important special case with its own specifics and lots of practical applications, so it potentially requires different methods from those used in the general case of ML for RDBs (see the next paragraph). Note that ML researchers and practitioners generally **do not think of single-table data as a special case of RDBs**, as evidenced by there being much more research on ML for single-table data (that does not mention RDBs at all) than on ML for RDBs. And our work aims to highlight that even if tabular data is not stored in an RDB, graph ML methods still can be effectively applied to it.

The second difference concerns how the data is represented after being converted to a graph. Since in our setting the source of data is a single table containing entities of a single type, the resulting graph is homogeneous. In contrast, in ML for RDBs, the source of data is multiple tables, each containing entities of its own type, and thus the resulting graph is heterogeneous, i.e., it has nodes and edges of different types. Different methods have been developed for working with homogeneous and heterogeneous graphs. For instance, there are a lot of specialized GNNs designed specifically for heterogeneous graphs (HAN, GTN, RSHN, HetGNN, MAGNN, HetSANN, HGT, etc.), many of which are very different from standard GNNs that are typically designed with homogeneous graphs in mind (GCN, GraphSAGE, GAT, GIN, etc.). Since the ML methods used for homogeneous and heterogeneous graphs can be very different, we believe that separate benchmarks for graphs of these two types are very useful. While GNNs for heterogeneous graphs can be applied to homogeneous graphs as well, it does not mean that they are the optimal choice for such graphs, so homogeneous graph benchmarks are needed to answer the question of what is the optimal choice, among others.

Finally, we would like to emphasize that due to these differences our work provides a meaningful contribution to both tabular and graph ML fields. In the last paragraph of the introduction section, we explain that our benchmark can serve two main purposes: “First, as discussed above, there are many real-world applications where graph structure can be naturally added to tabular data, and those interested in such applications can test their models on our benchmark. Hence, we hope that the proposed benchmark and new insights obtained using it will lead to a wider adoption of graph machine learning methods to tabular data in industry and science. Second, our benchmark provides an alternative testbed for evaluating GNN performance and, compared to standard graph benchmarks, offers datasets with very different feature types and prediction tasks which are more realistic and meaningful.” Regarding the first purpose, we show that graph ML methods can be effectively used for tabular data that is not stored in RDBs (while RDB benchmarks naturally cannot serve this purpose). As for the second purpose, we provide an alternative benchmark for evaluating GNNs that are not meant to work with heterogeneous graphs (which is the majority of modern GNNs), while RDB benchmarks can only be used to evaluate GNNs designed specifically for heterogeneous graphs.

Overall, our benchmark targets usecases that are significantly different from those of RDB benchmarks. That being said, both ML for RDBs and our work aim to bring graph ML methods to tabular data, which we believe to be potentially a very fruitful direction.
</content>
</thread_object>

</thread>


Metareview:
metareview:
While the topic of bridging graph and tabular machine learning is an important and emerging area, the paper under review fails to meet the bar for publication at ICLR due to the following concerns. 

Lack of Novelty: The proposed benchmark does not introduce significant methodological or conceptual advancements. The idea of constructing graph datasets from tabular data is well-explored in prior literature. Furthermore, as pointed out by multiple reviewers, the paper's contributions are largely incremental compared to existing benchmarks like RelBench and 4DBInfer.

Problem Setting and Scope: The distinction between single-table datasets with graph structure and relational databases is not convincingly motivated. The responses from the authors fail to justify why the proposed setting warrants an entirely new benchmark, as opposed to being an extension or a special case of relational database benchmarks.

Analysis and Generalization: The work does not provide a standardized mechanism for constructing graphs from tabular data, leaving the benchmark limited to the datasets provided. Additionally, the lack of exploration into broader task types, such as link prediction or multi-label classification, further restricts the generality and impact of the contribution.

Writing and Presentation: The manuscript is verbose in parts, with insufficient emphasis on crucial distinctions between the proposed work and related benchmarks. Important discussions are relegated to the appendix, diminishing their accessibility.

In summary, while the effort to create a comprehensive benchmark is appreciated, the paper lacks sufficient novelty, fails to convincingly establish the uniqueness of its setting, and is limited in scope. These shortcomings outweigh the strengths of the experimental rigor and practical dataset contributions. I recommend rejection.