Title: Statistical Inference for Gradient Boosting Regression
Venue: NeurIPS 2025
Accepted: yes
Keywords: Boosting

regression

nonparametric

trees

inference

statistics
Abstract: Gradient boosting is widely popular due to its flexibility and predictive accuracy. However, statistical inference and uncertainty quantification for gradient boosting remain challenging and under-explored. We propose a unified framework for statistical inference in gradient boosting regression. Our framework integrates dropout or parallel training with a recently proposed regularization procedure called Boulevard that allows for a central limit theorem (CLT) for boosting. With these enhancements, we surprisingly find that \textit{increasing} the dropout rate and the number of trees grown in parallel at each iteration substantially enhances signal recovery and overall performance. Our resulting algorithms enjoy similar CLTs, which we use to construct built-in confidence intervals, prediction intervals, and rigorous hypothesis tests for assessing variable importance in only $O(nd^2)$ time with the Nyström method. Numerical experiments verify the asymptotic normality and demonstrate that our algorithms perform well, do not require early stopping, interpolate between regularized boosting and random forests, and confirm the validity of their built-in statistical inference procedures.


Peer reviews:
comments: SUMMARY:
The paper builds upon Boulevard regularization and proposes two modified algorithm (dropout, parallel). They show the asymptotic normality of the proposed estimators including statistical inference.

STRENGTHS AND WEAKNESSES:
Strengths
- the paper is well motivated and tackles a relevant problem
- the paper proposes a unified framework: algorithm, proof of asymptotic normality and statistical inference
- the results and the corresponding assumptions are clearly discussed

Weaknesses
- In my opinion, once the asymptotic normality is achieved, the contributions regarding statistical inference are straightforward. It is not clear to me why the proposed two algorithms should be preferred over vanilla Boulevard regularization which is also asymptotically normal and allows for statistical inference. A clear description about the advantages and disadvantages (e.g. MSE, asymptotic efficiency, resources...) of vanilla Boulevard regularization, BRAT-D and BRAT-P including a discussion when to use what would be beneficial. Furthermore, to empirically compare the confidence intervals etc. with those based on vanilla Boulevard regularization, would strengthen the contribution, particularly Corollary 4.

QUESTIONS:
- The asymptotic variance is estimated. Can you please comment on the convergence to the true asymptotic variance?
- The proposed confidence intervals are quite large limiting the practical use. Could you please comment on potential optimality?
- It is hard to compare the MSE-performance of the different methods in Figure 2. Please make the figures larger (in particular Fig. 2 and 4).

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-06-25

comments: SUMMARY:
The paper proposes a variation of gradient boosting decision trees where dropout and parallel boosting are incorporated. It then proposes theoretical analysis of the algorithm showing central limit theorems (CLTs). This can be used for statistical inference: prediction, confidence, and reproduction intervals. Some numerical experiment results are presented.

STRENGTHS AND WEAKNESSES:
The paper proves for the first time a central limit theorem (CLT) for gradient boosting regression trees. However, I find the paper quite difficult to follow because of its dense mathematical presentation. While the theoretical derivations and equations seem rigorous and technically sound, their practical implications are not clearly conveyed. The central limit theorem (CLT) results rely on multiple assumptions, many of which seem to difficult to verify/ensure in practice. Moreover, the empirical evaluation does not convincingly demonstrate a significant performance improvement over strong baselines such as XGBoost. As a result, the contribution of the proposed method remains unclear in practice. Also, these CLT results are shown for a modified version of gradient boosting. Why this has not been shown for a traditional version of Friedman's gradient boosting is not clear.

QUESTIONS:
XGBoost and other tree ensembles have been widely used for many practical problems without CLT and statistical inference. Why does one need these in practice? I think the paper needs to motivate more the importance of statistical inference.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 2
IMPACT: 1
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nGradient boosting is one of the most widely used machine learning (ML)  methods, but the theoretical properties are less well understood than for other ML methods. The paper wants to fill this gap and provides theoretical results for boosting, namely a central limit theorem for boosting predictions. Two algoritms (Boulevard Regularized Additive regression Trees - Droput (BRAT-D) and BRAT parallel (BRAT-P) are proposed and their theoretical results are dervied. Finally, numerical experiments are presented which underscore the theoretical results.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**  \nOverall, the paper is an important contribution for understanding boosting algorithms better. The developed theory is very involved and impressive. The paper is structured clearly.\n\n**Weaknesses**\n- The literature review seems incomplete. The claim that \"Still, this is only method available for frequentist inference for gradient boosting,...\" is not correct. E.g. Luo et al. (2016) analyse gradient boosting with univariate regressions as base learners and derive theoretical properties. A comparison with this method would be interesting and  including other related papers in the literature review.\n\nYe Luo, Martin Spindler, Jannis Kueck (2016): High-Dimensional L_2Boosting: Rate of Convergence. arxiv 1602.08927, Journal of Machine Learning Research forthcoming.\n\n- A crucial point of boosting is early stopping and when to determine when to stop. This is also key for empirical applications, in particular having a feasible stopping criterion. It seems that the stopping time is given in the algorithms, but how would the optimal (feasible) stopping be determined and how does the stopping criterion affect the theoretical results? How does stopping / stopping criteria affect the empirical results?  \n-  The theoretical settings seems to be very much inspired by Random Forests, in particular \"parallel structure\", but are not directly related to the concrete implementation of gradient, in particular sequential proceeding.\n\nQUESTIONS:\n- Is it possible to give (feasible) stopping criteria for the boosting algorithms and analyze the behaviour under stopping criteria.\n- How do different stopping criteria affect the results from the numerical experiments?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on gradient boosting and builds upon the Boulevard regularization framework introduced by Zhou and Hooker (2022). It extends Boulevard in two directions: first, by introducing random dropout of trees; and second, by integrating the leave-one-out procedure and Boulevard regularization into a parallel boosting framework. The paper studies the statistical properties of both extensions. In particular, it shows that despite the randomness introduced by dropout, the algorithm still converges to a kernel ridge regression estimator that satisfies a central limit theorem (CLT). Moreover, the dropout improves signal recovery over the original Boulevard regularization. The authors also conduct theoretical analysis for the second algorithm, proving convergence to kernel ridge regression and a central limit theorem. The paper discusses confidence interval construction, and provides numerical experiments to support the theoretical claims.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper is very well written, with clear explanations of both the algorithms and the theoretical developments. \n- The paper makes both algorithmic and theoretical advances. On the algorithmic side, it introduces two meaningful extensions to Boulevard regularization, dropout boosting and parallel tree growth with leave-one-out structure, with improvement in signal recovery. On the theoretical side, the authors rigorously prove that these new algorithms retain key statistical properties, including convergence to kernel ridge regression and central limit theorems. \n- One of the key contributions is the introduction of dropout in the boosting process. While dropout is often used heuristically in ensemble methods, this paper provides a formal justification for its use and shows that it leads to improved signal recovery compared to the original Boulevard method. This insight is valuable both from a theoretical and practical standpoint.\n\nWeaknesses:\n- While the role of dropout is mentioned, not much discussion is on how to choose the dropout probability $p$ in practice.\n\nQUESTIONS:\n- Would it be possible to add some visual diagnostics (e.g., QQ plots) to verify the empirical normality implied by the CLT?\n- Could the authors provide a more in-depth discussion on how to tune hyperparameters such as the regularization strength $\\lambda$ and the dropout probability $p$ in practice? Practical guidance or heuristics for selecting these values would be valuable for applying the method effectively.\n\nLIMITATIONS:\nThe paper is theoretical and methodological in nature and does not raise direct societal concerns.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-02"}