Title: Regression Trees Know Calculus
Venue: NeurIPS 2025
Accepted: yes
Keywords: regression tree

gradient learning

interpretability

variable importance
Abstract: Regression trees have emerged as a preeminent tool for solving real-world regression problems due to their ability to deal with nonlinearities, interaction effects and sharp discontinuities. In this article, we rather study regression trees applied to well-behaved, differentiable functions, and determine the relationship between node parameters and the local gradient of the function being approximated. We find a simple estimate of the gradient which can be efficiently computed using quantities exposed by popular tree learning libraries. This allows tools developed in the context of differentiable algorithms, like neural nets and Gaussian processes, to be deployed to tree-based models. To demonstrate this, we study measures of model sensitivity defined in terms of integro-differential quantities and demonstrate how to compute them for regression trees using the proposed gradient estimates. Quantitative and qualitative numerical experiments reveal the capability of gradients estimated by regression trees to improve predictive analysis, solve tasks in uncertainty quantification, and provide interpretation of model behavior.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper suggests a method to estimate derivatives/gradients for sufficiently large regression trees. On this basis, estimators which transfer the integrated gradient method and the active subspace method from neural networks to regression trees are developed. The experiments show applicability for local explainability of model predictions and promising results in different settings related to the active subspace method, in particular regarding active subspace estimation.\n\nSTRENGTHS AND WEAKNESSES:\nQuality: From what I can tell, the submission seems technically sound, and proofs are provided for the theoretical results. I appreciate that a variety of applications/experiments was shown, and found especially the study on local explainability of model predictions on MNIST to be intriguing. The results regarding active subspace estimation are also convincing. It is clear from the experiments and the discussion that much effort was expended; substantial additional material is contained in the Appendix.\n\nClarity: I found the paper to be mostly clear overall. The introduction of the concepts/methods, including the ones from literature (integrated gradient, active subspace), is generally well done. However, there is also potential for improvement. I am not sure why the “2” in the central equation (3) is needed. Figure 2 is not properly referenced, only in the caption of Figure 3, even though it greatly helps to understand how the method works. Figure 3 should be made bigger and introduced a bit later; currently, it is almost impossible to analyze as it is so small, and most of the notation explained is not yet introduced at this point in the paper. The theorems in Section 4 require assumptions which are only given in the Appendix, and I suggest to at least give a hint of what these assumptions are in the main body of the paper. Section 5.2 should reference to Appendix B.1 where the datasets for Table 1 are introduced. In addition, I was not able to assess the supplementary material (when extracting, it consists of a single file without file ending). There are also minor grammatical, spelling and consistency issues (inconsistent capitalization, repeated introduction of abbreviations etc.) that should be rectified.\n\nSignificance: The introduction does a good job in clarifying the motivation of the work and the background highlights important related concepts/methods. I was well able to understand the rationale behind the paper and am convinced that a gradient approximation for regression trees can be of great value and can advance understanding in a meaningful way. I also find both the applicability for local explainability as well as for uncertainty quantification to be highly intriguing. I am unsure about the feasibility of the assumptions for the theorems in Section 4, however.\n\nOriginality: With its novel methods, the paper seeks to improve understanding of regression trees and enable additional applications for this method, which I find to be very interesting as it may prove to be complementary to research on neural networks. A lot of implications and potential for future work are highlighted in the paper. The overall reasoning for originality is convincing. While a brief discussion of SHAP is provided in the paper, I would be interested in an examination of the relation to BSHAP (Sundararajan & Najmi 2020).\n\nMukund Sundararajan, Amir Najmi (2020). Proceedings of the 37th International Conference on Machine Learning, PMLR 119:9269-9278.\n\nIn sum, I found the paper to be a good read and wish the authors all the best for their work on this interesting topic.\n\nQUESTIONS:\n1. As stated above, I am unsure about the feasibility of the assumptions for the (central!) theorems in Section 4 and the description/explaination is very brief. Could you elaborate whether they are typically fulfilled? Are there some clear conditions under which the assumptions are certain to be fulfilled? What is an example for not fulfilling the assumptions?\n2. As stated above, I would be interested in an examination of the relation to BSHAP (Sundararajan & Najmi 2020). From my understanding, this work (amongst others) discusses how splits in tree-based models encode gradient-like behaviors, which is a similar form of local explainability as well. \n3. The advantage of TBAS over the competing approaches seems to diminish for higher depth of the regression tree (cf. Table 1). Is this true and what is the reason for it?\n\nIf these questions are addressed satisfactorily, I may increase my evaluation score.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a simple method to estimate the gradient of a target function in a piecewise-constant regression tree, as per Algorithm 1. This is used to approximate the Integrated Gradient and Active Subspace Method as per sec 2.3. A variety of experiments indicates the utility of this method for regression trees.\n\nSTRENGTHS AND WEAKNESSES:\nTitle: \"Regression Trees Know Calculus\" is a rather provocative title, attributing self-awareness (\"know\") to regression trees. A  more academic title would be something like \"Obtaining and Exploiting Gradient Information from Regression Trees\".\n\nEq 3: \\mu^r_i and \\mu^l_i have not been defined!\n\nSec 4: I did not check the proofs in detail (Appendix A).\n\nSec 5.1 (IG): Re Fig 4. IG requires two points x and x^*. It is not explained what these are in your experiments. I guess x is the current digit (e.g. the 0 in left most panel), but what is x^*?\nLacking this information I do not find Fig 4 useful. Also the gradient is estimated at all pixels, not just the black ones (anyway, as I recall MNIST pixel values are real-valued). Why are the values only shown for the black pixels, or are the other values near 0? It would be better to show the IG values per pixel without the digit in the background.\n\nAlso presumably this target function has only 2 values (e.g. +1/-1 for the two classes) -- so is a gradient actually well defined here?\n\nSec 5.2 Table 1. I appreciate the efforts to indicate similar performance, as per the caption \"bold indicates confidence interval overlaps with the lowest confidence interval\". But confidence interval at what level (alpha?). Also, bear in mind that you are using the same test set so can do *paired comparisons*. Overall Table 1 suggests that TBAS improves over the identity and PCA rotations. Please also refer to Appendix B.1 and Fig 8 from the main text, as additional information is provided there.\n\nMore broadly, Rainio, O., Teuho, J. & Klen, R., Evaluation metrics and statistical tests for machine learning. Sci Rep 14, 6086 (2024). https://doi.org/10.1038/s41598-024-56706-x provides a useful flowchart (Fig 3) and sttaistical tests for the comparison of different methods. \n\nSec 5.3: Please explain what the function you are estimating is in the main text. (It is given in B.2). Also define how you compute the \"active subspace error\" (perhaps angle in radians, cosine of angle??)\n\n* Other points\n\nPlease distinguish between textual and paranthetical citations\nproperly. e.g. l 103 should be (Constantine, 2015) not as written, and\nthere are many other examples.  I recommend natbib (e.g. \\citet, \\citep)\nfor doing this.\n\nl 112 easy-to-compute\n\nl 169: \\bold{a} is undefined. And why change notation from x to z?\n\nl 286: second principal component -> second active subspace component?\n\nFig 9. This figure is not comprehensible without the text in sec C.1. The graphics and caption together should *explain* the figure, so you need to explain what the colour coding means (angle between true and\nestimated gradient)in the figure. Check other figures similarly. \n\nQuality: Good. Although I did not check the asymptotic analysis results in detail (sec 4), I find these results credible. The experiments are quite well done, but see comments above re significance of differences\nin Table 1.\n\nClarity: Good. I found the paper quite well written, except for a few gaps highlighted above.\n\nSignificance: Good: Despite the high profile of (deep) neural networks, it is true that decision tree methods are heavily used in data science applications, and thus advancements for regression trees could have\nsignificant impact.  To me the strongest result in the experiments is Table 1, and the improvements made by TBAS rotation.\n\nOriginality: Good: AFAIK there has not been prior work on estimating derivatives in regression trees. Alg 1 is almost embarassingly simple -- it is probably what you would come up with if someone forced\nyou to estimate a gradient from a piecewise-constant regression trees. But still, it seems not to have been done before, and to be useful, and there is an asymptotic analysis of its properties.\n\nQUESTIONS:\nTable 1: Please comment on the issue of the significance of differences, given my comments above.\n\nSec 5.1: Please clarify how x^* is chosen in these experiments. Also in Fig 4, should there be IG values for all pixels?\n\nGood answers to these questions would increase my overall score a bit.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-19"}

comments: SUMMARY:
This paper proposes a method to estimate the gradient of the underlying function from a regression tree estimator. With this estimation, one can improve the interpretation of the tree models.

STRENGTHS AND WEAKNESSES:
Strength: The overall idea is an important extension of widely used regression tree models. The proposed algorithm is simple and intuitive. The authors have demonstrated the usefulness of this idea with a couple of applications. The paper is well-written -- I enjoy reading it a lot. 

Weakness: The theoretical guarantees are not very satisfactory. 
Looking into Assumption 1 in the supplementary material, it seems that Theorem 4.1 relies on an assumption that the mean of f(x) in a give leaf converges at square-root rate. This assumption itself need to be proved for many tree-fitting algorithms. For example, if one use CART to fit the tree, then the splitting threshold itself will be data-dependent, and one cannot directly claim the square-root rate from CLT.

QUESTIONS:
1. Theorem 4.1 shows a high-probability convergence result for a given point x. Is it possible to prove a similar conclusion that holds uniformly for all points in the domain, assuming the derivatives of f have some smoothness. 

2. The intuition of this proposed method relies on the fact that the a deep tree is fitted. However, in practice, when trees are used in boosting or random forest, each individual tree would usually not be very deep. Could the authors give a comment on the performance of the proposed method in that setting?

LIMITATIONS:
See weakness and questions.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-06-21

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a method for estimating gradients from regression trees by computing finite differences between adjacent nodes in the tree structure. The authors propose using these gradient estimates to enable gradient-based interpretability techniques for tree models, specifically focusing on Active Subspace methods and Integrated Gradients. The key insight is that the difference in mean responses between left and right splits, normalized by node width, approximates partial derivatives. The paper demonstrates applications including tree-based active subspace estimation (TBAS), tree-based integrated gradients (TBIG), and shows these can improve predictive performance and provide model interpretation.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n- The core idea of extracting gradient information from tree structures is genuinely innovative and opens up new possibilities for tree-based interpretability.\n- The paper provides asymptotic analysis (Theorems 4.1, 4.2) establishing consistency of the gradient estimators under reasonable assumptions.\n- The method enables tree models to leverage existing gradient-based tools from other ML domains, potentially bridging interpretability techniques across model classes.\n- The gradient estimation can be computed in a single tree traversal, making it computationally attractive.\n- The evaluation covers multiple applications (active subspace estimation, integrated gradients, predictive improvement) across various datasets.\n\n### Weaknesses\n- Limited Scope of Application: The method requires \"sufficiently deep\" trees on \"well-behaved, differentiable functions.\" This significantly restricts its applicability since trees are often valued precisely for handling discontinuous or pathological data.\n- Questionable Fundamental Assumption: The paper's premise that regression trees can meaningfully approximate gradients of smooth functions contradicts why trees are typically chosen - their ability to handle non-smooth, discontinuous relationships.\n- Weak Empirical Results: Table 1 shows modest improvements that often fall within confidence intervals. The benefits don't strongly justify the additional complexity.\n- Incomplete Evaluation:\n- - No comparison to established tree interpretability methods (SHAP, permutation importance)\n- - Limited analysis of when the method fails or produces poor gradient estimates\n- - Missing evaluation on truly discontinuous functions\n- Presentation Issues:\n- - Heavy notation makes the core simple idea unnecessarily complex\n- - Some claims are overstated (e.g., \"first such linear sensitivity metric for trees\")\n- - The connection between gradient quality and tree depth could be made more explicit\n\n- Theoretical Limitations: The convergence rate of \\sqrt(P N^(-1) log N) may be slow in practice, and the assumptions (differentiability, sufficient depth) are quite restrictive.\n\nQUESTIONS:\n- Can you better characterize when your gradient estimates are reliable? What happens on discontinuous functions where trees typically excel? Consider providing failure cases or bounds on estimation quality.\n- How do your gradient estimates compare to true gradients on known smooth functions? Can you show convergence behavior as tree depth increases?\n- What depth and sample size requirements are needed for reliable gradient estimation in different dimensions? Can you provide practical guidance for practitioners?\n- How does TBAS compare to established tree interpretability methods like SHAP values or permutation importance?\n- How does the method perform with categorical variables, missing data, or truly non-smooth functions that motivate tree usage?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}