Title: Tree Ensemble Explainability through the Hoeffding Functional Decomposition and TreeHFD Algorithm
Venue: NeurIPS 2025
Accepted: yes
Keywords: explainability

gradient boosted trees

random forests

functional ANOVA decomposition

Hoeffding decomposition

Shapley values
Abstract: Tree ensembles have demonstrated state-of-the-art predictive performance across a wide range of problems involving tabular data. Nevertheless, the black-box nature of tree ensembles is a strong limitation, especially for applications with critical decisions at stake. The Hoeffding or ANOVA functional decomposition is a powerful explainability method, as it breaks down black-box models into a unique sum of lower-dimensional functions, provided that input variables are independent. In standard learning settings, input variables are often dependent, and the Hoeffding decomposition is generalized through hierarchical orthogonality constraints. Such generalization leads to unique and sparse decompositions with well-defined main effects and interactions. However, the practical estimation of this decomposition from a data sample is still an open problem. Therefore, we introduce the TreeHFD algorithm to estimate the Hoeffding decomposition of a tree ensemble from a data sample. We show the convergence of TreeHFD, along with the main properties of orthogonality, sparsity, and causal variable selection. The high performance of TreeHFD is demonstrated through experiments on both simulated and real data, using our treehfd Python package (https://github.com/ThalesGroup/treehfd). Besides, we empirically show that the widely used TreeSHAP method, based on Shapley values, is strongly connected to the Hoeffding decomposition.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper develops a method to approximate the Hoeffding Functional Decomposition (HFD) of tree ensembles. The authors first derive the HFD for tree ensembles with the property that the resulting functionals are piecewise constant. They then show interesting theoretical properties of this HFD including a near hierarchical orthogonality and provide an efficient way of estimating this HFD for tree ensembles.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n-------------------\n- Very interesting and important idea.\n- I enjoyed reading the paper, concepts are well explained and the proofs are overall easy to follow.\n- The idea of approximate hirarchical orthogonality is very nice, as is the fact that the Hoeffding decomposition can be estimated relatively efficiently\n- Great Illustrations and Experiments\n\nWeaknesses\n------------------\n- My biggest concern are about details of the formalization: HFD based on Cartesian Tree Partitions are a smart move, but once we think about consistency of the approach, that somehow means to me that there is a limiting partitioning towards we converge, i.e. something like a limiting tree ensemble. I provide more details below, but if I am not wrong about this, I think this needs to be discussed.\n\nQUESTIONS:\nAs mentioned above, I wonder whether consistency in Theorem 7 is not very oversimplified. The theorem and proof make sense to me for a fixed partition, but as far as I understand these partitions depend on the trees and these in turn change with the sample size. So basically the HFD for tree ensembles changes with n and it is not even clear to me what the limit should be. So it seems to me that the problem is more complex than what is formalized in Theorem 7. From a statisticians perspective, I would either need more explanation here if I am wrong, or a discussion on what kind of simplyfying assumptions were taken. I think with trees it is absolutely ok to first proof constistency results in a simplified setting.\n\nIf this can be addressed sufficiently, I am happy to increase my grade.\n\nOther than that I only have a few comments:\n- I do not understand the first equality in Lines 780 and 781 (Proof of Theorem 3). Is there a short sentence that can be added to explain this?\n- At the end of that proof I am missing the step where we go from \\eta_{\\ell} to \\eta, in fact the last equations after line 786 seem to suddenly jump from \\eta to \\eta_{\\ell}. It seems like a small step, but I think it should be explained.\n- It could be good to have some description of how we get from L^* to L_n on page 6. The first part is easy to see, but the expression in L_n after the \"+\" appears mysterious at this point.\n- Maybe a naive question from someone who does not understand computational complexity well: Why does the computational complexity not depend on n and p?\n- Line 353 \"hierarchical orthogonality\" --> \"hierarchical orthogonal\"\n- Detail \"under the same assumptions *than* Theorem 3\" sounds a bit weird do me. Maybe \"as in Theorem 3\"?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce TreeHFD, an algorithm estimating the Hoeffding Functional Decomposition (HFD) of tree ensembles in cases where only data samples are available. HFD decomposes a function prediction into a sum of lower-order functions over variable subsets and thereby naturally enables explainability through the main effects and low-order interactions. This approach is applied by the authors to tree ensembles with unknown input distribution by discretising the orthogonality constraints. Besides theoretical analysis of the method and resulting algorithm, the authors conduct simulations and experiments with TreeHFD and establish a connection to the functional decomposition provided by TreeSHAP.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- Interesting novel application of the HFD to tree ensembles in the practically relevant case of an available data sample but unknown input data distribution\n- Extensive theoretical analysis of the HFD for tree ensembles incl. the convergence of the proposed algorithm\n- Extensive experiments on both simulated and real-world data, including comparisons to relevant competitive approaches\n\nWeaknesses\n- Intuitive example for the application of treeHFD is missing. Given that it presents an XAI method to make effects of a black-box model more transparent, an example with real world data with large p and resulting interpretations would showcase its practical relevance.\n- No description of the difference between interventional and observational SHAP is provided. Although is only a compared method, this would be valuable to understand the experimental section better.\n- SHAP can be used to provide local explanations for a given data sample. TreeHFD could also evaluate the global functional decomposition at given inputs, however, exemplary evaluations of this scenario are not presented.\n- No details on the computational complexity of the algorithm are discussed (running times, scaling to higher-order interactions etc.). Similarly, it is not discussed in which scenarios treeHFD fails.\n\nQUESTIONS:\n- The whole model development and experimental setting revolves around regression. However, tree ensembles provide also SOTA prediction methods in the classification setting. To what extent can treeHFD be also applied/extended to this use case?\n- Could treeHFD also be used for feature selection? Could it, for example, be employed to prune noisy trees with many irrelevant variables?\n- Figure 2: Why are no orange points from the respective estimation with TreeSHAP included?\n- Table 2: Why are there NA values for the orthogonality of the datasets \"Power Plant\" and \"Superconduct.\"?\n- Housing Dataset: The authors write that TreeShap does not detect the peak, however, I would argue that it does (just to lower extent than treeHFD). Can the authors elaborate on this observation? Also, how would the main effects for TreeSHAP without interactions look like in this example?\n\nLIMITATIONS:\nPartially. While the authors claim in the checklist that they are discussed in Section 4 with the experiments, I cannot find them there. Instead I would have preferred a separate statement on the current limitations of TreeHFD and in which scenarios it fails.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces TreeHFD, a post-hoc explanation method for tree ensemble models that estimates the Hoeffding Functional Decomposition (HFD) directly from data without requiring knowledge of the input distribution. HFD breaks the model prediction into a sum of orthogonal components over variable subsets, enabling interpretable representations of main effects and interactions. TreeHFD ensures key properties like uniqueness, sparsity, near-orthogonality, and causal variable selection. It outperforms TreeSHAP in terms of stability and interpretability, and the authors show empirically that TreeSHAP closely approximates the HFD but suffers from entangled and noisy components.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper studies an important aspect in explainability by fining the Hoeffding functional decomposition for dependent data. The Hoeffding functional decomposition is a functional decomposition with extra constraints of zero-mean functions and orthogonality; this paper puts forward a way to construct such a decomposition for tree ensembles in polynomial time, which is a drastic improvement over the recursive construction in exponential time. That is why the paper is of significant importance. \n\nA key limitation of TreeHFD is its scalability with respect to the complexity of the trees in the ensemble.  Although the algorithm is efficient in terms of sample size, the complexity of the decomposition grows with the tree depth, the number of interaction terms, and the number of splits, which can become computationally intensive for deep or large ensembles unless pruning is applied. To keep the decomposition tractable, the authors restrict the order of interactions to at most two by setting the interaction hyperparameter d_I =2, meaning that only main effects and pairwise interactions are modeled. This setting seems to be consistently used across all experiments in the paper. While this promotes interpretability and computational efficiency, it may overlook higher-order interactions that could be relevant in some domains, and is in contrast with the efficiency claims in the paper. \n\nIn addition, the paper assumes that input features lie within the unit hypercube $[0,1]^p$ with a density bounded away from zero and infinity. While this assumption simplifies the theoretical analysis—ensuring well-behaved integrals, uniqueness of the decomposition, and control over orthogonality—it does not naturally hold in many real-world datasets where features vary in scale or distribution. Although normalization can map features to the $[0,1]$ range, this transformation may not have been used in the original trained model and   could obscure relationships in the original feature space. Moreover, the assumption of a uniformly bounded density may be unrealistic in high-dimensional or highly skewed settings, potentially limiting the robustness of the decomposition and its convergence guarantees.\n\nAnother limitation lies in the way the paper assumes the tree ensemble prediction function is constructed as a simple sum of the outputs from individual trees, i.e., $T = \\sum_\\ell T_\\ell$. While this assumption holds in some models like random forests (up to averaging), it does not directly apply to boosting algorithms like XGBoost, where each tree output is scaled by a learning rate. This assumption appears throughout the main theorems and derivations, yet the paper does not explicitly clarify whether these scaling factors are absorbed into the definition of $T_\\ell$ or how they impact the theoretical validity of the decomposition. Clarifying this point seems to be essential for a broader applicability of the paper. \n\nFor readers unfamiliar with the Hoeffding Functional Decomposition (HFD), the paper could benefit from a brief exposition of the classical case—where input features are assumed to be independent—and the recursive formulation used to construct the decomposition. This would not only make the paper more accessible to a broader audience, but also highlight the difficulty of constructing HFD even for independent features. \n\nI am willing to raise my score if these concerns are adequately addressed.\n\nQUESTIONS:\nThe paper’s focus is restricted to regression problems, where the output variable is continuous and modeled as $Y = m(X) + \\epsilon$. How could it be applied to the classification tasks?\n\nLIMITATIONS:\nTreeHFD operates directly on the structure of the trained tree ensemble and thus requires full access to the internal components of each tree, including split variables, split thresholds, and leaf values. It is not a huge limitation in explainability, but should be briefly discussed.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces TreeHFD, an algorithm designed to compute the Hoeffding Functional Decomposition (HFD) for tree ensemble models such as gradient boosted trees and random forests. The HFD is an explainability method that provides a unique decomposition of a function into a sum of main effects (univariate functions) and interaction terms (multivariate functions). However, its practical estimation, particularly in the common machine learning setting with dependent input variables and an unknown data distribution, is a challenging task. The authors propose an algorithm that takes a trained tree ensemble and a dataset as input, and formulate a loss function for each tree composed of two parts: a mean squared error term to ensure the decomposition accurately reconstructs the tree's predictions, and a penalty term to enforce the hierarchical orthogonality that defines pure interactions. This objective is then solved as a standard least-squares problem. The authors provide theoretical justification for their approach, namely the uniqueness of their proposed TreeHFD, the near-orthogonality of the final components, the sparsity of the decomposition, the algorithm's convergence, and its ability to perform causal variable selection. They also empirically demonstrate that TreeHFD accurately estimates the HFD on synthetic data.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The authors define the HFD specifically for tree ensembles, introducing a \"Cartesian tree partition\" input space splitting, which helps with the discretization of the orthogonality constraints.\n- The proposed work is grounded on mild assumptions, namely that the input distribution has bounded density on a hyperrectangle support, which is often a reasonable assumption for tree-based models.\n- The proposed TreeHFD algorithm is not merely a heuristic. The authors define a TreeHFD by adapting the classic HFD framework to the piecewise-constant nature of tree-based models. They formally derive its essential properties, such as uniqueness, orthogonality, sparsity, and causal variable selection.\n- The experimental evaluation is well-structured. The use of an analytical case with a known ground-truth HFD provides a clear demonstration of TreeHFD's accuracy. The qualitative analysis of the California Housing dataset coincides with common sense, although it might benefit from more analysis on the other variables and their interactions.\n\nWeaknesses:\n- The algorithm's computational cost is dependent on the number of cells in the Cartesian tree partition, which can increase exponentially with tree depth. The authors propose using hyperparameters ($d_T$, $d_V$) to prune deep trees or limit the number of variables considered as a practical solution. This approach, however, feels somewhat ad-hoc. The paper uses XGBoost as an ensemble (where the default depth is 6) , but for other ensembles that require deep trees, such as those in a standard Random Forest, this pruning could lead to accuracy degradation.\n\nQUESTIONS:\n1. Please expand the analysis on at least one or two more datasets, showing how TreeHFD captures main effects and interactions.\n\nLIMITATIONS:\nMentioned in the weaknesses:\nThe algorithm's computational cost is dependent on the number of cells in the Cartesian tree partition, which can increase exponentially with tree depth. The authors propose using hyperparameters ($d_T$, $d_V$) to prune deep trees or limit the number of variables considered as a practical solution. This approach, however, feels somewhat ad-hoc. The paper uses XGBoost as an ensemble (where the default depth is 6) , but for other ensembles that require deep trees, such as those in a standard Random Forest, this pruning could lead to accuracy degradation.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}