RelBench: A Benchmark for Deep Learning on Relational Databases
Authors: name: Joshua Robinson; name: Rishabh Ranjan; name: Weihua Hu; name: Kexin Huang; name: Jiaqi Han; name: Alejandro Dobles; name: Matthias Fey; name: Jan E. Lenssen; name: Yiwen Yuan; name: Zecheng Zhang; name: Xinwei He; name: Jure Leskovec
Published: 2024-07-29 14:46:13

Abstract: We present RelBench, a public benchmark for solving predictive tasks over
relational databases with graph neural networks. RelBench provides databases
and tasks spanning diverse domains and scales, and is intended to be a
foundational infrastructure for future research. We use RelBench to conduct the
first comprehensive study of Relational Deep Learning (RDL) (Fey et al., 2024),
which combines graph neural network predictive models with (deep) tabular
models that extract initial entity-level representations from raw tables.
End-to-end learned RDL models fully exploit the predictive signal encoded in
primary-foreign key links, marking a significant shift away from the dominant
paradigm of manual feature engineering combined with tabular models. To
thoroughly evaluate RDL against this prior gold-standard, we conduct an
in-depth user study where an experienced data scientist manually engineers
features for each task. In this study, RDL learns better models whilst reducing
human work needed by more than an order of magnitude. This demonstrates the
power of deep learning for solving predictive tasks over relational databases,
opening up many new research opportunities enabled by RelBench.