ICML 2025

Compressing Tree Ensembles through Level-wise Optimization and Pruning
Laurens Devos * 1 2 Timo Martens * 1 2 Deniz Can Oruc¸ 1 2 Wannes Meert 1 2 Hendrik Blockeel 1 2 Jesse Davis 1 2
Abstract
Tree ensembles (e.g., gradient boosting decision
trees) are often used in practice because they offer
excellent predictive performance while still being
easy and efficient to learn. In some contexts, it
is important to additionally optimize their size:
this is specifically the case when models need
to have verifiable properties (verification of fair-
ness, robustness, etc. is often exponential in the
ensemble’s size), or when models run on battery-
powered devices (smaller ensembles consume less
energy, increasing battery autonomy). For this
reason, compression of tree ensembles is worth
studying. This paper presents LOP, a method for
compressing a given tree ensemble by pruning or
entirely removing trees in it, while updating leaf
predictions in such a way that predictive accuracy
is mostly unaffected. Empirically, LOP achieves
compression factors that are often 10 to 100 times
better than that of competing methods.
1. Introduction
Ensembles of decision trees, a.k.a. decision forests, are
frequently used in practice because of their ease of training,
excellent overall performance, and high efficiency. Their
training requires little or no hyperparameter tuning, the
computational cost of both training and prediction is very
small, and for tabular data, they are usually among the best
performing models (Grinsztajn et al., 2022).
This paper is about compressing decision forests: reducing
their size as measured by, e.g., the total number of leaves
in the forest, while maintaining their accuracy. In the light
of the above successes, one might wonder whether this is
worthwhile: if forests are already so efficient, can we gain
*Equal contribution
1KU Leuven Department of Computer
Science, Leuven, Belgium 2Leuven.AI, KU Leuven Institute for
Artificial Intelligence, Leuven, Belgium.
Co...