NRGBoost: Energy-Based Generative Boosted Trees
Authors: name: João Bravo
Published: 2024-10-04 15:54:02

Abstract: Despite the rise to dominance of deep learning in unstructured data domains,
tree-based methods such as Random Forests (RF) and Gradient Boosted Decision
Trees (GBDT) are still the workhorses for handling discriminative tasks on
tabular data. We explore generative extensions of these popular algorithms with
a focus on explicitly modeling the data density (up to a normalization
constant), thus enabling other applications besides sampling. As our main
contribution we propose an energy-based generative boosting algorithm that is
analogous to the second order boosting implemented in popular packages like
XGBoost. We show that, despite producing a generative model capable of handling
inference tasks over any input variable, our proposed algorithm can achieve
similar discriminative performance to GBDT on a number of real world tabular
datasets, outperforming alternative generative approaches. At the same time, we
show that it is also competitive with neural network based models for sampling.