Title: On the Convergence of CART under Sufficient Impurity Decrease Condition
Venue: NeurIPS 2023
Accepted: yes
Keywords: decision tree

CART
Abstract: The decision tree is a flexible machine-learning model that finds its success in numerous applications. It is usually fitted in a recursively greedy manner using CART. In this paper, we study the convergence rate of CART under a regression setting. First, we prove an upper bound on the prediction error of CART under a sufficient impurity decrease (SID) condition \cite{chi2020asymptotic} -- our result is an improvement over the known result by \cite{chi2020asymptotic} under a similar assumption. We show via examples that this error bound cannot be further improved by more than a constant or a log factor. Second, we introduce a few easy-to-check sufficient conditions of the SID condition. In particular, we show that the SID condition can be satisfied by an additive model when the component functions satisfy a ``locally reverse Poincare inequality". We discuss a few familiar function classes in non-parametric estimation to demonstrate the usefulness of this conception.

On the Convergence of CART under Sufficient Impurity Decrease Condition

Abstract

The decision tree is a flexible machine learning model that finds its success in numerous applications. It is usually fitted in a recursively greedy manner using CART. In this paper, we investigate the convergence rate of CART under a regression setting. First, we establish an upper bound on the prediction error of CART under a sufficient impurity decrease (SID) condition [10] - our result improves upon the known result by [10] under a similar assumption. Furthermore, we provide examples that demonstrate the error bound cannot be further improved by more than a constant or a logarithmic factor. Second, we introduce a set of easily verifiable sufficient conditions for the SID condition. Specifically, we demonstrate that the SID condition can be satisfied in the case of an additive model, provided that the component functions adhere to a 'locally reverse Poincaré inequality". We discuss several well-known function classes in non-parametric estimation to illustrate the practical utility of this concept.

1 Introduction

The decision tree [5] is one of the most fundamental and popular methods in the machine learning toolbox. It utilizes a flowchart-like structure to recursively partition the data and allows users to derive interpretable decisions. It is usually constructed in a data-dependent greedy manner, with an algorithm called CART [5]. Thanks to its computational efficiency and ability to capture nonlinear structures, decision trees have served as the foundation for various influential algorithms for ensemble learning, including bagging [6], random forest [7] and gradient boosting [11]. These algorithms are among the best-known nonparametric models for supervised learning with tabular data [14].

Despite its remarkable empirical success in various applications, our theoretical understanding of decision trees remains somewhat limited. The data-dependent splits employed in CART pose challenges for rigorous theoretical analysis. While some easy-to-analyze variants of CART have been widely studied [18, 4, 3], rigorous theoretical analysis of the original version of CART has been absent until recently. In particular, a recent line of work [19, 24, 10, 16] has established the consistency and prediction error bounds for CART and random forest. See Section 1.1 for a comprehensive review of related literature.

In this paper, we study the decision tree under a nonparametric regression model:

with x i = ( x (1) i , ..., x ( p ) i ) ∈ R p , y i ∈ R ; ϵ i is the noise with zero-mean, and f ∗ ( x ) is the signal function. We consider a random design where { x i } n i =1 are i.i.d. from some underlying distribution µ supported on [0 , 1] p , and { ϵ i } n i =1 are i.i.d. independent of { x i } n i =1 . For an estimator ˆ f of f ∗ , we use

the L 2 prediction error ∥ ˆ f -f ∗ ∥ 2 L 2 ( µ ) as a measure of the estimator ˆ f . Consider a regression tree algorithm that produces a heuristic (not necessarily exact) solution ˆ f of

where T d is the space of binary (i.e. each splitting node has two children) axis-aligned regression trees with maximum depth d . The statistical performance of the estimator ˆ f , therefore, is affected by the following three factors: (i) Approximation error: how accurately can f ∗ be approximated by a regression tree in T d . (ii) Estimation error: the variance of the estimator, which is affected by the complexity of the function space T d . A larger space T d (i.e. larger d ) can reduce the approximation error, but it may also increase the estimation error and lead to overfitting. (iii) Optimization error: the discrepancy between ˆ f and the exact optimal solution of the optimization problem (2). Indeed, if ˆ f is the exact optimal solution of (2) (i.e. optimal regression tree), it can approximate a broad class of f ∗ [9], and the prediction error can be analyzed via the classical theory of least-squares estimators [22, 13]. However, since solving (2) to optimality is computationally challenging for large-scale data, heuristics like CART are typically employed in practice. To understand the statistical performance of CART, the major difficulty lies in the analysis of the optimization error of CART.

As a greedy algorithm, CART examines only one split in each iteration (see Section 1.3 for details). Although decision trees are intended to capture multivariate structures across different features, the greedy nature of CART poses challenges in approximating functions f ∗ that have complex multivariate structures. For example, consider the two-dimensional "XOR gate" function f ∗ ( u ) := 1 { u ∈ [0 , 1 / 2) 2 } +1 { u ∈ [1 / 2 , 1) 2 } [14], and assume that x i has a uniform distribution on [0 , 1] 2 . Note that f ∗ itself is a depth-2 decision tree, with (e.g.) the root node split with the first coordinate at 1 / 2 , and the left and right children split with the second coordinate at 1 / 2 . However, due to the symmetry structure of f ∗ , when the sample size n is large, the root split of CART is completely fitting to noises and is likely to generate a split near the boundary [5, 8]. Consequently, the resulting estimator ˆ f fails to capture the true tree structure encoded in f ∗ . The major difficulty in fitting the XOR gate function with CART lies in the complicated interaction between the two coordinates.

The discussions above naturally lead to the following two questions about CART:

What conditions on f ∗ are needed such that CART can approximate f ∗ and yield a consistent estimator?

If f ∗ satisfies such conditions, what is the convergence rate of the prediction error of CART?

In this paper, we aim to address these fundamental questions by providing sufficient conditions on f ∗ such that one can establish convergence rates of CART. These sufficient conditions are quite flexible to include a broad class of nonparametric models. In the following, we first discuss some related literature and the basics of CART. In Section 2, we prove a general error bound for CART under a sufficient impurity decrease (SID) condition [10] of f ∗ . This error bound is an improvement over the existing results under similar conditions [10]. In Section 3, we introduce a sufficient condition for the SID condition on f ∗ , and use it to show that the SID condition can be satisfied by a broad class of f ∗ .

1.1 Related literature

The inception and analysis of decision trees and CART algorithm can be traced back to the seminal work of Breiman [5]. Breiman's study marked the first significant step towards establishing the consistency of CART, albeit under stringent assumptions concerning tree construction. The original formulation of CART, characterized by data-dependent splits, poses challenges when it comes to rigorous mathematical analysis. To address this issue, subsequent research efforts [18, 4, 3] have introduced alternative variants of CART that lend themselves to more tractable theoretical examinations. For instance, certain studies [12, 4, 3, 2] assume random selection of splitting features in the tree. Other approaches incorporate random selections of splitting thresholds [12, 2], while some restrict splits to occur exclusively at the midpoint of the corresponding cell in the chosen coordinate [3]. These simplifications ensure the independence between the splitting rules of the tree and the training data - the tree depends on the data only via the estimation in each cell. Such independence simplifies the analysis of the resultant trees. In a similar vein, alternative strategies have

been proposed, such as the employment of a two-sample approach [23] to compute splitting rules and cell estimations separately, giving rise to what is known as an 'honest tree". This methodology also ensures independence between various aspects of the tree construction and estimation process.

Since the first consistency result in [5], the strict theoretical analysis of the original CART algorithm has remained elusive until recent advancements. A significant breakthrough came with the seminal work of Scornet et al. [19], which demonstrated the consistency of CART under the assumption of an additive model [15] with continuous component functions. It makes a technical contribution on the decrease of optimization error of CART. This technical argument strongly relies on the additive model. Subsequent works [17, 16] further extended the theoretical analysis by providing non-asymptotic error bounds for CART under a similar additive model. However, the rates derived in [17, 16] is a slow rate of O ((log( n )) -1 ) , which seems not easy to be improved under the same assumption [8]. Although the additive model is a well-studied model in statistical learning, it is not immediately evident why such a modeling assumption is specifically required for CART to function effectively.

Another line of work [24, 10] adopts more intuitive assumptions on the signal function f ∗ to ensure the progress of CART. In particular, Chi et al. [10] introduced a sufficient impurity decrease (SID) condition. In essence, the SID condition posits that for every rectangular region B within the feature space, there exists an axis-aligned split capable of reducing the population variance of f ∗ (conditional on B ) by a fixed ratio. Note that such a split cannot be directly identified by CART, as it necessitates knowledge of the population distribution of x i . Instead, CART employs empirical variance of y i n i =1 to guide its cell splitting decisions. Nonetheless, the SID condition is a strong assumption on the approximation power of tree splits, which can ensure the consistency of CART via an empirical process argument [10]. A similar condition has also appeared implicitly in [24]. Under the SID condition, Chi et al. [10] studied the consistency and convergence rates of CART, demonstrating a polynomial convergence O ( n -γ ) of the prediction error. A possible limitation of the SID condition is the difficulty of verification: it necessitates the decreasing condition to hold true for all rectangles. Although a few examples are provided in [10], the general procedure for verifying whether a specific f ∗ satisfies this condition remains unclear. Notably, its relationship with the additive model assumption in [19, 17] remains to be fully understood.

Lastly, it is worth mentioning that several recent studies [20, 8] have investigated the performance lower bounds of CART, shedding light on its limitations. Specifically, Tan et al. [20] demonstrated that even when assuming an additive regression function f ∗ , any single-tree estimator still suffers from the curse of dimensionality. Consequently, these estimators can only achieve convergence rates considerably slower than the oracle rates established for additive models [21]. Another study by Cattaneo et al. [8] delved into the splitting behavior of CART and provided theoretical support for empirical observations [5] indicating that CART tends to generate splits near boundaries in cases where the signal is weak. These lower bounds support the importance of using trees in ensemble models to reduce the prediction error. Although we focus on analyzing CART for a single tree in this paper, the result can be adapted to the analysis of ensembles of trees for the fitting of more complex function classes.

1.2 Contributions

The first contribution of our paper is a refined analysis of CART under the SID condition. Although the convergence of CART has been studied in [10] under SID condition, the analysis of [10] seems not tight when the noises have light-tails (see discussions in the Appendix C). We establish an improved analysis of CART under this setting, and show by examples that our error bound is tight up to log factors.

The second contribution of this paper is the decoding of the mystery of the SID condition. We discuss a sufficient condition under which the SID condition holds true. In particular, we introduce a class of univariate functions that satisfy an locally reverse Poincaré inequality , and show that additive functions with each univariate component being in this class satisfy the SID condition. This builds a connection between the two types of assumptions in the literature: additive model and SID condition. We discuss a few examples that how the locally reverse Poincaré inequality can be verified.

1.3 Basics of CART

We review the methodology of CART to build the tree. The primary objective of decision trees is to find optimal partitions of feature space that produce a minimal variance of response variables. CART constructs the tree and minimizes the variance using a top-down greedy approach. It starts with a root node that represents the whole feature space R p - this can be viewed as an initial tree with depth 0 . At each iteration, CART splits all the leave nodes in the current tree into a left child and a right child; the depth of the tree increases by 1 , and the number of leave nodes in the tree doubles. At each leave node, it takes an axis-aligned splits with maximum variance (impurity) decrease. Let E be the set of all intervals (which can be open, closed, or open on one side and closed on another side). Define the set of rectangles in [0 , 1] p as:

For each A ∈ A , define I A := { i ∈ [ n ] | x i ∈ A } . For a leave node that represents a rectangle A ∈ A , the impurity decrease of a split in feature j ∈ [ p ] with threshold b ∈ R is given by:

where y I := 1 |I| ∑ i ∈I y i for any I ⊆ [ n ] , and

To split on A , CART takes ˆ j and ˆ b that maximize the impurity decrease, i.e.,

This splitting procedure is repeated until the tree has grown to a given maximum depth d .

1.4 Notations

For a, b > 0 , we write a = O ( b ) if there exists a universal constant C > 0 such that a ≤ Cb ; we write b = Ω( a ) if a = O ( b ) , and write a = Θ( b ) if both a = O ( b ) and a = Ω( b ) are true. For values a n,p , b n,p that may depend on n and p , we write a n,p = ˜ O ( b n,p ) if a n,p = O ( b n,p log γ ( np )) for some fixed constant γ ≥ 0 .

2 Error bound of CART under SID property

We study the prediction error bound of CART for the regression problem (1). We focus on a random design as in the following assumption.

Assumption 2.1 (i) (Random design) Suppose { x i } n i =1 are i.i.d. random variables with a distribution µ supported on [0 , 1] p . Suppose µ has a density p X (w.r.t. Lebesgue measure) on [0 , 1] p satisfying 0 < ¯ θ ≤ p X ( u ) ≤ ¯ θ < ∞ for all u ∈ [0 , 1] p for some constants ¯ θ ≤ 1 and ¯ θ ≥ 1 .

(ii) (Error distribution) Suppose { ϵ i } n i =1 are i.i.d. zero-mean bounded random variables with | ϵ i | ≤ m< ∞ for some m> 0 . Suppose { ϵ i } n i =1 are independent to { x i } n i =1 .

(iii) (Bounded signal function) Suppose sup u ∈ [0 , 1] p | f ∗ ( u ) | ≤ M < ∞ for some constant M .

In Assumption 2.1 (i), it is assumed that the features x i are supported on [0 , 1] p for convenience. The same results (different by at most a constant) can be proved if we replace [0 , 1] p with an arbitrary bounded rectangle. The second assumption (Assumption 2.1 (ii)) requires the error terms { ϵ i } n i =1 to be i.i.d. bounded - this is the key difference of our paper and the assumptions made in [10]. In particular, [10] made a milder assumption on the error terms that allows them to have heavy tails. Their result, however, when specifying to the case of bounded noises, is not tight (see

discussions in Appendix C for a comparison). Note that for convenience we have assumed that ϵ i is bounded. If instead, we assume the noises are sub-Gaussian, similar conclusions can also be proved via a truncation argument (the bound may be different by a log factor). The third assumption (Assumption 2.1 (iii)) is a standard assumption for nonparametric regression models [22, 13].

Under the random design setting in Assumption 2.1, we are ready to introduce the sufficient impurity decrease condition on the underlying function f ∗ . Let X be a random variable in R p that has the same distribution as x 1 and is independent to { x 1 , ..., x n } . For any A ∈ A , j ∈ [ p ] and b ∈ R , define:

where A L and A R are defined in (5), and Var( f ∗ ( X ) | X ∈ A ) means the conditional variance defined as

It is not hard to see the similarity between ∆( A,j, b ) and ̂ ∆( A,j, b ) (defined in (4)), where the former can be viewed as a population variant of the latter. Intuitively, for a split with feature j and threshold b , ∆( A,j, b ) measures variance decrease of f ∗ after this split. The SID condition below assumes that there is a split with a large variance decrease.

Assumption 2.2 (Sufficient Impurity Decrease) There exists a constant λ ∈ (0 , 1] such that for all A ∈ A ,

Note that Assumption 2.2 is a condition depending on both the function f ∗ and the underlying distribution X ∼ µ . Briefly speaking, the SID condition requires that the best split must decrease the population variance by a constant factor (note that λ is required to be bounded away from 0 ). Intuitively, the condition (9) can be satisfied if f ∗ has a significant 'trend" along some axis in A , but may be hard to be satisfied if f ∗ is relatively 'flat" in A . Since Assumption 2.2 requires (9) being satisfied for all cells A ∈ A , it essentially requires that f ∗ is not 'too flat" on any local rectangle A . In Section 3, we develop a technical argument to translate this intuition into a rigorous statement, and use it to check Assumption 2.2 for a wide class of functions.

With Assumptions 2.1 and 2.2 at hand, we are ready to present the main result in this section. For any function g : R p → R , let ∥ g ∥ L 2 ( µ ) denote the L 2 -norm of g with respect to the measure µ .

Theorem 2.3 Suppose Assumptions 2.1 and 2.2 hold true. Let ̂ f ( d ) ( · ) be the tree estimated by CART with depth d . Suppose n is large enough such that 2 ¯ θe 2 d n ∨ log(72 p d ( n +1) 2 d /δ ) n < 3 / 4 . Then there exists a universal constant C > 0 such that with probability at least 1 -δ , it holds that for any α > 0 ,

where U := M + m . In particular, taking α = 1 /d and d = ⌈ log 2 ( n ) / (1 -log 2 (1 -λ )) ⌉ , it holds

where ϕ ( λ ) := -log 2 (1 -λ ) 1 -log 2 (1 -λ ) , and C λ,U is a constant that only depends on λ and U .

The error bound in (10) consists of two terms. The first term corresponds to the bias of the approximating f ∗ using CART. It decreases geometrically as d increases, which is suggested by the SID condition. The second term is O (2 d d log( np ) /n ) , which corresponds to the estimation variance of CART. Here the term 2 d represents the model complexity - a fully-grown depth d tree has 2 d leaves. The proof of Theorem 2.3 is presented in Appendix A. It is based on a careful technical argument that controls the difference between the population impurity decrease ∆( A,j, b ) and the empirical impurity decrease ̂ ∆( A,j, b ) . In particular, our analysis is different from that of [10]. Note that [10] makes a different assumption on the error distribution, and their technical argument is not sufficient for the proof of Theorem 2.3. Particularly, the error bound in (11) is an improvement of the results proved in Theorem 1 of [10]-see Section C for a detailed comparison.

Note that the convergence rate in (11) has a crucial dependence on the coefficient λ . For a large value of λ , the exponent ϕ ( λ ) is also larger, leading to a faster convergence rate in (11). On the other end, if λ is small, as λ → 0 , we have ϕ ( λ ) → 0 , and the convergence rate in (11) can be arbitrarily slow. Nonetheless, as long as the SID condition is satisfied with λ bounded away from 0 , (11) shows a polynomial rate, which is better than the logarithmic rate in [17, 16] without the SID condition.

To explore the tightness of the error bounds in Theorem 2.3, we consider a simple example with f ∗ being a univariate linear function.

Example 2.1 (Univariate linear function) Suppose p = 1 , and x i are i.i.d. from the uniform distribution on [0 , 1] . Suppose f ∗ is a univariate linear function: f ∗ ( t ) = t . Then the SID condition (Assumption 2.2) is satisfied with coefficient λ = 3 / 4 . As a result, it holds ϕ ( λ ) = 2 / 3 , and the error bound in the RHS of (11) is ˜ O ( n -2 / 3 ) .

Proof. For any interval A = [ ℓ, r ] , we have

Take b := ( ℓ + r ) / 2 , then we have

Combining (12) and (13) we know that Assumption 2.2 is satisfied with λ = 3 / 4 . □

As shown by Example 2.1, when f ∗ is a univariate linear function and { x i } n i =1 are from a uniform distribution, the error bound in (11) reduces to ˜ O ( n -2 / 3 ) . This rate matches the lower bound for any partition-based estimator (see e.g. [13] Chapter 4), 2 hence showing that the upper bound in (11) cannot be improved (by more than a log factor) via any refined analysis. To verify the rate ˜ O ( n -2 / 3 ) , we conduct a simulation shown by Figure 1. In particular, Figure 1 presents the log-log plot of the prediction error of CART versus different sample sizes, where the depth d is taken as stated above (11). A reference line of slope -2 / 3 is also shown. It can be seen that the slope of the line for CART is also roughly -2 / 3 , which is consistent with the rate ˜ O ( n -2 / 3 ) derived in Example 2.1.

7.0

7.5

8.0

8.5

9.0

9.5

10.0

10.5

log(

n

)

8.5

8.0

7.5

7.0

6.5

f

f

*

||

2

2

)

CART

reference line

3 Function classes satisfying SID condition

In this section, we introduce a class of sufficient conditions that facilitate the easy verification of the SID condition for a broad range of functions. Specifically, we focus on the case when f ∗ has an additive structure:

where f ∗ j 's are univariate functions. Note that additive model has also been assumed in prior works such as [19, 17, 16] for the study of CART. If we only assume that each component function f ∗ j has bounded total variation, then only a slow rate O (log -1 ( n )) is known [17]. In the following, we discuss a few sufficient conditions concerning the component functions f ∗ j under which f ∗ satisfies the SID condition, consequently guaranteeing a faster convergence rate as indicated by Theorem 2.3.

First, we introduce a key class of univariate functions that satisfy a special integral inequality.

Definition 3.1 (Locally Reverse Poincaré Class) For a univariate differentiable function g on an interval Q ⊂ R , we say it belongs to the Locally Reverse Poincaré (LRP) class on Q with parameter τ if for any subinterval [ a, b ] ⊆ Q ,

We use the notation LRP ( Q,τ ) to denote the class of all such functions.

It is worth noting that for any given univariate differentiable function g and a fixed interval [ a, b ] , there exists a constant τ such that the reverse of inequality (15) holds true, as established by the Poincaré inequality. The condition (15) requires a uniform constant τ > 0 such that the reverse of the Poincaré inequality holds true. Note that the LHS of (15) is equivalent to the square of the total variation of g on [ a, b ] , and the RHS is a constant multiple of the conditional variance of g on [ a, b ] (under the uniform distribution). Intuitively, this condition can be satisfied when the function g exhibits a clear 'trend" within any interval [ a, b ] rather than being fuzzy fluctuations. We give a few examples below.

Example 3.1 (Strongly increasing function) Let g be a strictly increasing function on [0 , 1] with c 2 ≥ g ′ ( t ) ≥ c 1 > 0 for all t ∈ [0 , 1] . Then g ∈ LRP ([0 , 1] , 2 √ 3 c 2 /c 1 ) .

Example 3.1 provides a direct generalization of the simple linear function showcased in Example 2.1. Note that this example was also discussed in [10]. In Example 3.1, since g ′ ( t ) is consistently bounded away from zero, verifying the inequality 15 becomes straightforward. Below we present a more nuanced example where g ′ ( t ) may equal zero at certain point t ∈ [0 , 1] .

Example 3.2 (Smooth and strongly convex function) Let g be a L -smooth and σ -strongly-convex function on [0 , 1] for some L > σ > 0 , i.e.,

for all t, s ∈ [0 , 1] . Then g ∈ LRP ([0 , 1] , 110( L/σ ))

Note that for a smooth and strongly convex function g , the gradient g ′ ( t ) can be zero at some t ∈ [0 , 1] . But the strong convexity condition guarantees that there is at most one point t where g ′ ( t ) = 0 , and the inequality (15) can still be satisfied. Additional details can be found in Appendix B.

Example 3.3 (Polynomial with degree r ) There exists a constant C r that only depends on r such that for any univariate polynomial g of degree at most r , g ∈ LRP ( R , C r ) .

Although the set of polynomials (with a degree at most r ) appears to be complicated, it is a finitedimensional linear space. Therefore the differentiation is a bounded linear operator between finite dimensional normed linear spaces. As a result, the conclusion of Example 3.3 can be proved via a variable transformation argument. See Appendix B for details.

Examples 3.1 - 3.3 demonstrate that the LRP condition can be readily verified for various well-known function classes, but the relationship between the LRP class and the SID condition remains unclear. Below we present the first main result in this section, which establishes a connection between the LRP class and the SID condition under the additive model.

Proposition 3.1 Suppose Assumption 2.1 holds true, and f ∗ has an additive structure as in (14) . If f ∗ k ∈ LRP ([0 , 1] , τ ) for all k ∈ [ p ] , then Assumption 2.2 is satisfied with λ = 4 ¯ θ/ ( pτ 2 ¯ θ ) .

As shown by Proposition 3.1, for the additive model (14), when the basic conditions in Assumption 2.1 are satisfied, the SID condition holds true for f ∗ as long as each component function is in an LRP class. Particularly, the SID condition is satisfied for component functions from Examples 3.1 - 3.3. Note that the coefficient λ in Proposition 3.1 is proportional to 1 /p , which implies that it can be very small when p is large. In particular, with this λ = Θ(1 /p ) (with ¯ θ, ¯ θ and τ being constants), by Theorem 2.3, the prediction error bound in the RHS of (11) is O ( n -c/p ) for some constant c > 0 3 . Since the exponent is dependent on p , the curse of dimensionality becomes evident. This seems to indicate that the analysis of Proposition 3.1 could potentially be refined. However, given the negative results demonstrated by [20, 8], it is likely that for tree-based estimators, even with the additive model, the curse of dimensionality cannot be circumvented.

Proposition 3.1 requires each component of the additive function to lie in an LRP class. Actually, this condition can be further relaxed as in the following proposition.

Proposition 3.2 Suppose Assumption 2.1 holds true, and f ∗ has an additive structure as in (14) . Given integer r ≥ 1 and constants α > 0 , β ≥ 1 . Suppose for each k ∈ [ p ] there exist 0 = t ( k ) 0 < t ( k ) 1 < · · · < t ( k ) r -1 < t ( k ) r = 1 with t ( k ) j -t ( k ) j -1 ≥ α/r and f ∗ k ∈ LRP ([ t j -1 , t j ] , β ) for all j ∈ [ r ] . Then Assumption 2.2 is satisfied with parameter λ = ¯ θ/ ( p ¯ θτ 2 ) where τ 2 = max { 2 r ¯ θ θ , r 2 2 α } max { 9 β 2 , 32 + β 2 } .

¯

By Proposition 3.2, an additive function f ∗ satisfies the SID condition if each component function is a piecewise function, with each piece belonging to an LRP class. Notably, continuity at the joining points of consecutive pieces is not necessary. However, for technical reasons, it is required that each piece has a length that is not excessively small, as determined by the parameter α . See Appendix B for the proofs of Proposition 3.1 and 3.2.

4 Proof sketch of Theorem 2.3

We provide a sketch of the proof of Theorem 2.3. For p ≥ 1 and d ≥ 1 , define

̸

That is, each rectangle in A p,d has at most d dimensions that are not the full interval [0 , 1] . When p ≥ d , A p,d contains all the rectangles in [0 , 1] p , i.e. A p,d = A . However, when d is much smaller than p (in a high-dimensional setting), A p,d contains much fewer rectangles than A . Note that for a decision tree with depth d , each leaf node represents a rectangle in A p,d .

The proof of Theorem 2.3 builds upon a few technical lemmas which provide uniform bounds between the empirical and populational quantities. These results may hold their own significance in the study of other partition-based algorithms. For δ ∈ (0 , 1) , define values

where ¯ θ is the constant in Assumption 2.1 ( i ) . Note that we have ¯ t ( δ ) ≤ O ( d log( np )+log(1 /δ ) n ) . We have the following uniform bounds on empirical and populational mean on every rectangle in A p,d .

Lemma 4.1 Suppose Assumption 2.1 holds true. Suppose ¯ t 2 ( δ/ 12) < 3 / 4 . Then with probability at least 1 -δ , it holds

For a tree-based estimator in practice, the prediction at each leaf node is usually given by the sample average of y i for all the samples i routed to that leaf, i.e., the value ¯ y A for a leaf with rectangle A . Therefore, Lemma 4.1 provides a uniform bound on the error if we replace the prediction at each leaf by the populational conditional mean E ( f ∗ ( X ) | X ∈ A ) . It is worth noting that the gap between ¯ y A and E ( f ∗ ( X ) | X ∈ A ) is rescaled by the quantity √ P ( X ∈ A ) . Intuitively, when the rectangle A is small and hence P ( X ∈ A ) is small, few points are routed to the rectangle A . As a result, the corresponding gap between ¯ y A and E ( f ∗ ( X ) | X ∈ A ) can be large - which leads to a large discrepancy of the estimate and the signal locally in this cell A . However, in another perspective, since the cell A is small, a large discrepancy in A only makes a small contribution to the overall error ∥ ̂ f ( d ) -f ∗ ∥ 2 L 2 ( X ) . Therefore, the factor √ P ( X ∈ A ) in the LHS of (19) is a proper balance of these two aspects. This is the key technical argument that differs from the analysis in [10]. It helps establish the tighter bounds in Theorem 2.3.

Lemma 4.2 Suppose Assumption 2.1 holds true. Given a constant α > 0 . Given any δ ∈ (0 , 1) , suppose ¯ t 2 ( δ/ 36) < 3 / 4 . Then with probability at least 1 -δ , it holds

for some universal constant C ′ > 0 .

Lemma 4.2 establishes upper bounds on the discrepancy between empirical and populational impurity decrease. It builds an avenue to translate the SID condition into an empirical counterpart, which is further used to derive the decrease of objective value.

With Lemmas 4.1 and 4.2 at hand, we are ready to wrap up the proof of Theorem 2.3. First, we have

where ˜ f ( k ) is a tree with the same splitting rule as ̂ f ( k ) but leaf predictions replaced by the populational means (in that leaf). The term J 2 ( k ) can be bounded as

where L ( k ) is the set of leaves of ̂ f ( k ) at depth k ; A t is the corresponding rectangle for a leaf t ; and C is a universal constant. The last inequality in (21) is by Lemma 4.1. The term J 1 ( k ) can be written as

Making use of Lemma 4.2 and by some algebra (see Appendix A for details), a recursive inequality can be established:

The proof of Theorem 2.3 is complete by telescoping the inequality above, and combining the upper bounds for J 1 ( k ) and J 2 ( k ) with (20).

5 Conclusion and discussions

We have proved an error bound on the prediction error of CART for a regression problem when f ∗ satisfies the SID condition. This error bound cannot be improved by more than a log factor. We have also discussed a few sufficient conditions under which we can show that an additive model satisfies the SID condition.

One possible limitation of this work is that: it seems that the SID coefficients λ derived in Section 3 are not tight. Since λ appeared in the exponent of the rate in Theorem 2.3, a tighter estimate of λ leads to an improved convergence rate in (11). We leave it as a future work for a more precise analysis of the SID condition.

References

Abhineet Agarwal, Yan Shuo Tan, Omer Ronen, Chandan Singh, and Bin Yu. Hierarchical shrinkage: Improving the accuracy and interpretability of tree-based models. In International Conference on Machine Learning , pages 111-135. PMLR, 2022.

Sylvain Arlot and Robin Genuer. Analysis of purely random forests bias. arXiv preprint arXiv:1407.3939 , 2014.

Gérard Biau. Analysis of a random forests model. The Journal of Machine Learning Research , 13(1):10631095, 2012.

Gérard Biau, Luc Devroye, and Gäbor Lugosi. Consistency of random forests and other averaging classifiers. Journal of Machine Learning Research , 9(9), 2008.

L Breiman, JH Friedman, R Olshen, and CJ Stone. Classification and regression trees. 1984.

Leo Breiman. Bagging predictors. Machine learning , 24(2):123-140, 1996.

Leo Breiman. Random forests. Machine learning , 45(1):5-32, 2001.

Mattias D Cattaneo, Jason M Klusowski, and Peter M Tian. On the pointwise behavior of recursive partitioning and its implications for heterogeneous causal effect estimation. arXiv preprint arXiv:2211.10805 , 2022.

Sabyasachi Chatterjee and Subhajit Goswami. Adaptive estimation of multivariate piecewise polynomials and bounded variation functions by optimal decision trees. The Annals of Statistics , 49(5):2531-2551, 2021.

Chien-Ming Chi, Patrick Vossler, Yingying Fan, and Jinchi Lv. Asymptotic properties of high-dimensional random forests. The Annals of Statistics , 50(6):3415-3438, 2022.

Jerome H Friedman. Greedy function approximation: a gradient boosting machine. Annals of statistics , pages 1189-1232, 2001.

Robin Genuer. Variance reduction in purely random forests. Journal of Nonparametric Statistics , 24(3):543562, 2012.

László Györfi, Michael Kohler, Adam Krzyzak, Harro Walk, et al. A distribution-free theory of nonparametric regression , volume 1. Springer, 2002.

Trevor Hastie, Robert Tibshirani, Jerome H Friedman, and Jerome H Friedman. The elements of statistical learning: data mining, inference, and prediction , volume 2. Springer, 2009.

Trevor J Hastie. Generalized additive models. In Statistical models in S , pages 249-307. Routledge, 2017.

J Klusowski and P Tian. Large scale prediction with decision trees. Journal of the American Statistical Association , 2022.

Jason M Klusowski. Universal consistency of decision trees in high dimensions. arXiv preprint arXiv:2104.13881 , 2021.

Yi Lin and Yongho Jeon. Random forests and adaptive nearest neighbors. Journal of the American Statistical Association , 101(474):578-590, 2006.

Erwan Scornet, Gérard Biau, and Jean-Philippe Vert. Consistency of random forests. The Annals of Statistics , 43(4):1716-1741, 2015.

Yan Shuo Tan, Abhineet Agarwal, and Bin Yu. A cautionary tale on fitting decision trees to data from additive models: generalization lower bounds. In International Conference on Artificial Intelligence and Statistics , pages 9663-9685. PMLR, 2022.

Zhiqiang Tan and Cun-Hui Zhang. Doubly penalized estimation in additive regression with highdimensional data. 2019.

Alexandre B Tsybakov. Nonparametric estimators. Introduction to Nonparametric Estimation , 2009.

Stefan Wager and Susan Athey. Estimation and inference of heterogeneous treatment effects using random forests. Journal of the American Statistical Association , 113(523):1228-1242, 2018.

Stefan Wager and Guenther Walther. Adaptive concentration of regression trees, with application to random forests. arXiv preprint arXiv:1503.06388 , 2015.

A Proof of Theorem 2.3

A.1 Notations

We first define some notations in the context of the model (1). For p ≥ 1 and d ≥ 1 , define

̸

That is, each rectangle in A p,d has at most d dimensions that are not the full interval [0 , 1] . Note that for a decision tree with depth d , each leave node represents a rectangle in A p,d . Furthermore, for δ ∈ (0 , 1) , define values

where ¯ θ is the constant in Assumption 2.1 ( i ) . Note that we have ¯ t ( δ ) ≤ O ( d log( np/δ ) /n ) .

For two values a, b > 0 , we write a ≲ b if there is a universal constant C > 0 such that a ≤ Cb . We write a ≲ r b if there is a constant C r that only depends on r such that a ≤ C r b .

A.2 Technical lemmas

Now we can introduce the major technical results to establish the error bound.

Lemma A.1 Suppose Assumption 2.1 holds true. Suppose ¯ t 2 ( δ/ 12) < 3 / 4 . Then with probability at least 1 -δ , it holds

The proof of Lemma A.1 is presented in Section A.4. Note that Lemma A.1 provides a uniform bound on the gap between the populational mean E ( f ∗ ( X ) | X ∈ A ) and the sample mean ¯ y I A . This is used to derive the geometric decrease of the bias, using the SID assumption.

Lemma A.2 Suppose Assumption 2.1 holds true. Given any δ ∈ (0 , 1) , suppose ¯ t 2 ( δ/ 4) < 3 / 4 . Then with probability at least 1 -δ it holds

The proof of Lemma A.2 is presented in Section A.5. Lemma A.2 provides a uniform deviation gap between the square root of probability and sample frequency over all sets in A p,d . Note that this uniform bound is stronger than a result without a square root (which can be obtained easily via Hoeffding's inequality and a union bound), and is useful to prove the final error bound in Theorem 2.3.

For any rectangle A ∈ A , j ∈ [ p ] and b ∈ R , define

We have the following identity regarding the impurity decrease of each split.

Lemma A.3 For any rectangle A ∈ A , j ∈ [ p ] and b ∈ R , it holds

Proof. We just present the proof of the second equality. The proof of the first equality can be proved similarly. Note that

For the first term, we have

Similarly, we have

The proof is complete by combining (26), (27) and (28).

□

Lemma A.4 Suppose Assumption 2.1 holds true. Given a constant α > 0 . Given any δ ∈ (0 , 1) , suppose ¯ t 2 ( δ/ 36) < 3 / 4 . Then with probability at least 1 -δ , it holds

Proof. For A ∈ A p,d -1 , j ∈ [ p ] and a ∈ R , by Lemma A.3 we have

Define the events E 1 and E 2 :

Then by Lemmas A.1 and A.2, we have P ( E i ) ≥ 1 -δ/ 3 for i = 1 , 2 , so we have P ( ∩ 2 i =1 E i ) ≥ 1 -δ . Below we prove (29) and (30) conditioned on the events E 1 and E 2 .

Note that

To bound J 1 , we have

where the second inequality is by event E 1 . Similarly, to bound J 3 , we have

To bound J 2 , note that

where the second inequality made use of the event E 2 . Combining (32) - (35), we have

which implies (by Young's inequality)

By a similar argument, we have

∆

R

(

A,j, a

)

(1 + 1

/α

)

·

2500

Summing up (36) and (37), and by (31), we have

This completes the proof of (29). The proof of (30) is by a similar argument.

Define a function

(37)

□

Lemma A.4 provides upper bounds between ∆( A,j, b ) and ̂ ∆( A,j, b ) , which serves as a link to translate the population impurity decrease to sample impurity decrease. With all these technical lemmas at hand, we are ready to present the proof Theorem 2.3, as shown in the next subsection.

A.3 Completing the proof of Theorem 2.3

Define events

Then by Lemmas A.1 and A.4, and note that from the statement of Theorem 2.3, ¯ t 2 ( δ/ 72) < 3 / 4 , so we have P ( E 1 ) ≥ 1 -δ/ 2 and P ( E 2 ∪ E 3 ) ≥ 1 -δ/ 2 , which implies P ( ∪ 3 i =1 E i ) ≥ 1 -δ . In the following, we prove (10) using a deterministic argument conditioned on ∪ 3 i =1 E i .

For any k ∈ [ d ] and any leave node t of ̂ f ( k ) (recall that ̂ f ( k ) is the decision tree by CART with depth k ), let A ( k ) t be the corresponding cube, that is, for any x ∈ R p , x ∈ A ( k ) t if and only if x is routed to t in ̂ f ( k ) . Let L ( k ) be the set of all leave nodes of ̂ f ( k ) . Then we have

where X n 1 is the set of iid random variables { x 1 , ..., x n } , and X is a random variable having the same distribution as x 1 but independent of X n 1 . In other words, ˜ f ( k ) is a tree with the same splitting structure as ̂ f ( k ) and replaces the prediction value of each leave node as the populational conditional mean of f ∗ ( · ) .

First, using Cauchy-Schwarz inequality, we have

To bound J 1 ( d ) , we derive recursive inequalities between J 1 ( k ) and J 1 ( k +1) for all 0 ≤ k ≤ d -1 . Note that

U

2

¯

t

(

δ/

36) + (1 +

α

)

|I

A

R

n

|

|

¯

y

I

A

-

¯

y

I

AR

|

2

For each t ∈ L ( k ) , let t L and t R be the two children of t , then we have

where

Let us define

Then we have

where the first inequality is by event E 3 , the second inequality is by the definition of ( ˆ j t , ˆ b t ) , and the third inequality is because of event E 2 . By Assumption 2.2, we have

Combining (42), (43) and (44), we have

Summing up the inequality above for all t ∈ L ( k ) , we have

Using the inequality above recursively for k = 0 , 1 , ...., d -1 , we have

To bound J 2 ( d ) , we have

where the inequality made use of event E 1 .

Using (45) and (46), and recalling (40), we have

This completes the proof of (10). To prove (11), by taking α = 1 /d and d = ⌈ log 2 ( n ) / (1 -log 2 (1 -λ )) ⌉ , we have

Note that for s = log 2 ( n ) / (1 -log 2 (1 -λ )) we have (1 -λ ) s = 2 s /n , hence by taking d = ⌈ log 2 ( n ) / (1 -log 2 (1 -λ )) ⌉ , we have

Combining (47), (48) and (49) and note that Var( f ∗ ( X )) ≤ M < U , we have

this completes the proof of (11).

A.4 Proof of Lemma A.1

The main idea of proving Lemma A.1 is to find a proper finite net of the set A p,d , control the gap on this net, and finally prove the result for all A ∈ A p,d based on the approximation gap of the net. We need a few auxiliary results. Let S := { 0 , 1 /n, 2 /n, ..., ( n -1) /n, 1 } , and define

For any A = ∏ p j =1 [ ℓ j , u j ] ∈ A p,d , define

where ℓ ′ j := max { s ∈ S | s ≤ ℓ j } , and u ′ j := min { s ∈ S | s ≥ u j } . Roughly speaking, A ′ is the smallest box with all edges in S that contains A . For any ˜ A = ∏ p j =1 [ ˜ ℓ j , ˜ u j ] ∈ ˜ A p,d with ˜ u j -˜ ℓ j ≥ 2 /n for all j ∈ [ p ] , define

̸

̸

and define B p,d to be the set of all such sets, that is

The following lemma can be easily verified from the definitions of ˜ A p,d and B p,d .

Lemma A.5 (1) For any A ∈ A p,d , there exists B ∈ B p,d such that A ′ \ A ⊆ B . (2) P ( X ∈ B ) ≤ 2 ¯ θd/n for all B ∈ B p,d .

(3) The cardinality

Finally, for any t ≥ 0 , we define

̸

Lemma A.6 Suppose Assumption 2.1 holds true. Let z 1 , ..., z n be i.i.d. bounded random variables with | z 1 | ≤ V < ∞ almost surely. Assume that for each i ∈ [ n ] , z i is independent of { x j } j = i , but may be dependent on x i . Given any δ ∈ (0 , 1) , with probability at least 1 -δ , it holds

where U = M + m .

Proof. For each fixed A ∈ ˜ A p,d \ ˜ A p,d ( ¯ t 1 ( δ )) , note that

so by Lemma D.1 with t = 2 V √ P ( X ∈ A ) √ ¯ t 1 ( δ ) , γ 2 = (2 V ) 2 P ( X ∈ A ) and b = 2 V , we have

where ( i ) is because P ( X ∈ A ) ≥ ¯ t 1 ( δ ) (since A ∈ ˜ A p,d \ ˜ A p,d ( ¯ t 1 ( δ )) ). As a result, we have

where the last inequality makes use of Lemma A.5 (3).

□

Lemma A.7 Let D be a finite collection of measurable subsets of [0 , 1] p satisfying P ( X ∈ D ) ≤ ¯ α for all D ∈ D (for some constant ¯ α ∈ (0 , 1) ). Given any δ ∈ (0 , 1) , if

then with probability at least 1 -δ it holds

Proof. For any fixed D ∈ D , denote α = P ( X ∈ D ) , then by Lemma D.2, for any t ∈ (0 , 3 / 4] , we have

where the third inequality makes use of Lemma D.3 and the assumption t ≤ 3 / 4 . Take t = w (¯ α, δ ) , and note that

we have

where the last inequality is because of the definition of w (¯ α, δ ) . Taking the union bound we have

Corollary A.8 Under Assumption 2.1 and given δ ∈ (0 , 1) , suppose ¯ t 2 ( δ ) < 3 / 4 , then with probability at least 1 -δ , it holds

Proof. Apply Lemma A.7 with D = B p,d and ¯ α = 2 ¯ θd/n , and note that |B p,d | ≤ ( n +1) 2 d p d (by Lemma A.5 (3)) and the definition ¯ t 2 ( δ ) = 2 ¯ θe 2 d n ∨ log( p d ( n +1) 2 d /δ ) n . □

̸

Lemma A.9 Suppose Assumption 2.1 holds true. Let z 1 , ..., z n be i.i.d. bounded random variables with | Z | ≤ V < ∞ almost surely. Assume that for each i ∈ [ n ] , z i is independent of { x j } j = i , but may be dependent on x i . Given any δ ∈ (0 , 1) , suppose ¯ t 2 ( δ/ 2) < 3 / 4 , then with probability at least 1 -δ , it holds

Proof. Define events E 1 and E 2 :

Then by Lemma A.6 and Corollary A.8, we have P ( E 1 ) ≥ 1 -δ/ 2 and P ( E 2 ) ≥ 1 -δ/ 2 , hence P ( E 1 ∩E 2 ) ≥ 1 -δ . Below we prove that when E 1 and E 2 hold true, inequality (50) holds true.

Note that for any A ∈ A p,d \ A p,d ( ¯ t ( δ/ 2)) ,

To bound T 1 , we have

where the second inequality makes use of Lemma A.5 (1), and the third inequality is by E 1 .

To bound T 2 , note that

where the inequality is by event E 2 and because A ′ ∈ ˜ A p,d and P ( X ∈ A ′ ) ≥ P ( X ∈ A ) ≥ ¯ t ( δ/ 2) ≥ ¯ t 1 ( δ/ 2) . Note that

where the first inequality is by Lemma A.5 (2); the second inequality is by the definition of ˆ t 2 ( δ/ 2) in (22); the third inequality is because A ∈ A p,d \ A p,d ( ¯ t ( δ/ 2)) . As a result of (53) and (54), we have

To bound T 3 , note that

The proof is complete by combining inequalities (51), (52), (55) and (56), and note that

Now we are ready to wrap up the proof of Lemma A.1.

Completing the proof of Lemma A.1

Define events E 1 and E 2 :

Then by Lemma A.9 with z i = y i and z i = 1 respectively, we know that P ( E i ) ≥ 1 -δ/ 4 for all i = 1 , 2 . So we know P ( ∩ 2 i =1 E i ) ≥ 1 -δ . Below we prove that inequality (23) is true when ∩ 2 i =1 E i hold.

Define a := 100 ¯ t ( δ/ 8) . Then it holds

On the other hand, for any A ∈ A p,d \ A p,d ( a ) , by event E 1 , we have

where the second inequality is because P ( X ∈ A ) ≥ a = 100 ¯ t ( δ/ 8) . Therefore we know ∑ n i =1 1 { x i ∈ A } > 0 , and we can write

By event E 2 , and note that a ≥ ¯ t ( δ/ 8) , we have

By event E 1 , and note that a ≥ ¯ t ( δ/ 8) , we have

Combining (59), (60) and (61), we have

Combining the inequality above with (57) we have

□

A.5 Proof of Lemma A.2

Define a := ¯ t ( δ/ 4) and b := a + 2 ¯ θd n . Define events E 1 and E 2 :

Then by Lemmas A.7 and A.9, we know that P ( E 1 ) ≥ 1 -δ/ 2 and P ( E 2 ) ≥ 1 -δ/ 2 , so P ( E 1 ∩ E 2 ) ≥ 1 -δ . Below we prove (24) when E 1 ∩ E 2 holds.

For A ∈ A p,d , if P ( X ∈ A ) ≤ a , then P ( A ′ ) ≤ a + 2 ¯ θd n = b . So we have

where the third inequality is by event E 1 and the definition of b ; the fourth inequality is because

As a result, we have

where the second inequality made use of (62).

On the other hand,

where the last inequality is by event E 2 .

Combining (63) and (64) the proof is complete.

B Proofs in Section 3

For any interval E ∈ E and any univariate function g on [0 , 1] , let V g ( E ) be the total variation of g on E . For the additive model (14) and a rectangle A = ∏ p j =1 E j ∈ A , define V f ∗ ( A ) = ∑ p j =1 V f ∗ j ( E j ) . Recall that X is a random variable with the same distribution as x i , and X ( j ) is the j -th coordinate of X .

B.1 Technical lemmas

Lemma B.1 For any rectangle A ⊆ [0 , 1] p , any j ∈ [ p ] and any b ∈ R , it holds

Proof. We use the notations ν := E ( f ∗ ( X ) | X ∈ A ) , ν L := E ( f ∗ ( X ) | X ∈ A L ) and ν R := E ( f ∗ ( X ) | X ∈ A R ) . First, note that

Similarly, we have

E ( ( f ∗ ( X ) -ν ) 2 1 { X ∈ A R } ) = E ( ( f ∗ ( X ) -ν R ) 2 1 { X ∈ A R } ) +( ν R -ν ) 2 P ( X ∈ A R ) (66)

Summing up (65) and (66) we have

Note that

Combining (67) and (68) we have

Lemma B.2 Suppose Assumption 2.1 holds true, and f ∗ has the additive structure in (14) . Then for any A = ∏ p j =1 [ ℓ j , u j ] ⊆ [0 , 1] p , it holds

where q ( k ) A ( t ) := P ( X ( k ) ≤ t | x 1 ∈ A ) .

Proof. For a fixed A = ∏ p j =1 [ ℓ j , u j ] ⊆ [0 , 1] p , without loss of generality, assume E ( f ∗ ( X ) | X ∈ A ) = 0 . Note that for any j ∈ [ p ] ,

where s the integration variable. Because q ( j ) A ( s ) = P ( X ∈ A L ( j, s )) / P ( X ∈ A ) , using Lemma B.1 and recall that we have assumed E ( f ∗ ( X ) | X ∈ A ) = 0 , we have

where the last equality makes use of the assumption that E ( f ∗ ( X ) | X ∈ A ) = 0 . Combining the inequality above with (69), we have

As a result, we have

By the additive structure (14) we have

Combining (70) and (71), the proof is complete.

□

Lemma B.3 Suppose Assumption 2.1 holds true, and f ∗ has the additive structure in (14) . If for any A = ∏ p j =1 [ ℓ j , u j ] ⊆ [0 , 1] p and any k ∈ [ p ] it holds

Then Assumption 2.2 is satisfied with λ = ¯ θ/ ( pτ 2 ¯ θ ) .

Proof. Given A = ∏ p j =1 [ ℓ j , u j ] ⊆ [0 , 1] p , without loss of generality, assume E ( f ∗ ( X ) | X ∈ A ) = 0 . Let p X ( · ) be the density of X on [0 , 1] p . Then we have

where the second inequality made use of Assumption 2.1 ( i ) . Denote c j := 1 u j -ℓ j ∫ u j ℓ j f ∗ j ( t ) dt and c := ∑ p j =1 c j , then we have

Combining the inequality above with (73) we have

We use H 2 k to denote the LHS of (72), then (72) implies

As a result of (74) and (75), we have

By Lemma B.2 we have

where the second inequality is by (76), and the last inequality made use of the Cauchy-Schwarz inequality.

B.2 Proof of Proposition 3.1

For any A = ∏ p j =1 [ ℓ j , u j ] ⊆ [0 , 1] p and any k ∈ [ p ] it holds

where the first inequality is by Cauchy-Schwarz inequality, and the second is because f ∗ k ∈ LRP ([0 , 1] , τ ) . Using Lemma B.3, the proof of complete.

B.3 Proof of Proposition 3.2

Then the conclusion follows Lemma B.3.

For fixed A and k ∈ [ p ] , to simplify the notation, we denote g := f ∗ k , a := ℓ k , b := u k , q ( t ) := q ( k ) A ( t ) for all t ∈ [ ℓ k , u k ] , and t j := t ( k ) j for j = 0 , 1 , ..., r . Then (77) can be written as

For any s ∈ (0 , 1) , define ∆ g ( s ) := lim t → s + g ( t ) -lim t → s -g ( t ) . Let j ′ , j ′′ ∈ [ r ] such that t j ′ -1 ≤ a < t j ′ and t j ′′ -1 < b ≤ t j ′′ , and define r ′ = j ′′ -j ′ +1 , and

Then we have

We have the following 4 claims bounding the terms in the last line of the display above.

Claim B.4 For j ∈ { 1 , r ′ } , it holds

□

Proof of Claim B.4 : We just prove the claim for j = 1 . The proof for j = r ′ follows a similar argument. To prove the claim for j = 1 , we discuss two cases.

where the second inequality is because g ∈ LRP (( a, z 1 ) , β ) ; and the last inequality makes use of the fact q ( z 1 ) ≤ ¯ θ ( z 1 -a ) / ( ¯ θ ( b -a )) .

(Case 2) q ( z 1 ) > 1 / 2 . Then we have

As a result,

where the first inequality is by Cauchy-Schwarz inequality; the second inequality is because g ∈ LRP (( a, z 1 ) , β ) ; the third inequality is by (81).

Combining (Caes 1) and (Case 2), the proof of Claim B.4 is complete.

Claim B.5 For j ∈ { 1 , r ′ -1 } , it holds

Proof of Claim B.5 : We just prove the claim for j = 1 . The proof for j = r ′ -1 follows a similar argument. To prove the claim for j = 1 , we discuss two cases.

(Case 1) | ∆ g ( z 1 ) | > 4 max { ∫ z 1 z 0 | g ′ ( t ) | d t , ∫ z 2 z 1 | g ′ ( t ) | d t } . Then by Lemma D.6 we have

Note that

So by (82) and (83) we have

As a result,

where the second inequality is by Cauchy-Schwarz inequality.

(Case 2) | ∆ g ( z 1 ) | ≤ 4 max { ∫ z 1 z 0 | g ′ ( t ) | d t , ∫ z 2 z 1 | g ′ ( t ) | d t } . Then we have

□

By the same argument in (80), we have

On the other hand,

where the second inequality is because g ∈ LRP (( z 1 , z 2 ) , β ) ; the last inequality is because z 2 -z 1 ≥ α/r ≥ ( α/r )( b -a ) . By (84), (85) and (86), we have

Combining (Case 1) and (Case 2), the proof of Claim B.5 is complete.

Claim B.6 For 2 ≤ j ≤ r ′ -1 , it holds

Proof of Claim B.6 : Note that

where the first inequality is by Cauchy-Schwarz inequality; the second inequality is because g ∈ LRP (( z j -1 , z j ) , β ) ; the last inequality is by the assumption that t j -t j -1 ≥ α/r .

□

Claim B.7 For 2 ≤ j ≤ r ′ -2 , it holds

Proof of Claim B.7 : We discuss two cases.

(Case 1) | ∆ g ( z j ) | > 4 max { ∫ z j z j -1 | g ′ ( t ) | d t , ∫ z j +1 z j | g ′ ( t ) | d t } . Then by Lemma D.6 we have

As a result,

□

where the second inequality is by Cauchy-Schwarz inequality; the third inequality is because g ∈ LRP (( z j -1 , z j ) , β ) and g ∈ LRP (( z j , z j +1 ) , β ) .

Combining (Case 1) and (Case 2), and note that b -a ≤ 1 , the proof of Claim B.7 is complete.

□

Completing the proof of Proposition 3.2

By (79) and note that

By Claims B.4 and B.6, we have

By Claims B.5 and B.7, we have

By (87), (88) and (89) we have

Hence (78) is true, and the proof of Proposition 3.2 is complete.

B.4 Proof of Example 3.1

Given [ a, b ] ⊆ [0 , 1] , without loss of generality, assume ∫ b a g ( t ) = 0 (because the infimum in w is achieved at w = ∫ b a g ( t ) ). Let t 0 ∈ [ a, b ] be the point with g ( t 0 ) = 0 . Since g ′ ( t ) ≥ c 1 > 0 , we have

Similarly,

As a result, we have

On the other hand, since | g ′ ( t ) | ≤ c 2 , we have

Combining (90) and (91), we have

B.5 Proof of Example 3.3

It suffices to prove that there exists a constant C r such that for any univariate polynomial with a degree at most r and for any a < b ,

We first prove the conclusion when a = 0 and b = 1 . Let P ( r ) be the set of all univariate polynomials with degree at most r . Note that P ( r ) is a finite-dimensional linear space, and the differential operator Φ : g ↦→ g ′ is a linear mapping on P ( r ) . As a result, there exists C r such that

for all g ∈ P ( r ) .

For general a < b , given g ∈ P ( r ) , define h ( s ) := g ( a +( b -a ) s ) , then h ∈ P ( r ) . So we have

Note that and

Combining (93), (94) and (95), we know that

where the last step is by Cauchy-Schwarz inequality.

B.6 Proof of Example 3.2

It suffices to prove that for any a, b ∈ [0 , 1] with a < b , it holds

Once (96) is proved, the conclusion is true via Jensen's inequality.

Below we prove (96). Denote C = L/ 10 . For given a, b ∈ [0 , 1] , without loss of generality, we assume the median of g on [ a, b ] is 0 , i.e., ∫ b a 1 { g ( t ) ≥ 0 } d t = ∫ b a 1 { g ( t ) < 0 } d t = ( b -a ) / 2 (otherwise translate by a constant). We discuss two different cases. We denote m := min t ∈ [ a,b ] {| g ′ ( t ) |} and M := max t ∈ [ a,b ] {| g ′ ( t ) |} .

(Case 1) m ≥ C ( b -a ) . Since g is L -smooth on [0 , 1] , we have

Hence we have where the second inequality is by the assumption of (Case 1). Since min t ∈ [ a,b ] {| g ′ ( t ) |} = m> 0 , without loss of generality, we assume that g ′ ( t ) > 0 for all t ∈ [ a, b ] . Denote t 0 = ( a + b ) / 2 . By our assumption that the median of g on [ a, b ] is 0 , we know that g ( t 0 ) = 0 . Since g is convex, for any t ∈ [ a, t 0 ] , we have

which implies g ( t ) ≤ t 0 -t t 0 -a ( g ( a ) -g ( t 0 )) ≤ 0 . As a result, we have

On the other hand,

Combining (98) and (99) we have

where the second inequality made use of (97).

(Case 2) m<C ( b -a ) . Then by the L -smoothness of g we have

so we have

Define interval [ t 1 , t 2 ] := { t ∈ [ a, b ] | g ( t ) ≤ 0 } . By our assumption that the median of g on [0 , 1] is 0 , we have t 2 -t 1 = ( b -a ) / 2 . Denote t 0 = argmin t ∈ [ a,b ] g ( t ) . Define function f on [ t 1 , t 2 ] :

Then 0 ≥ f ( t ) ≥ g ( t ) for all t ∈ [ t 1 , t 2 ] (because g is convex). Note that

As a result,

where the first and last inequalities are because 0 ≥ f ( t ) ≥ g ( t ) for all t ∈ [ t 1 , t 2 ] ; the second equality is by (101). Note that for any t ∈ [ t 1 , t 2 ] ,

where the first inequality is because g is σ -strongly-convex, and the second is because t 0 is the minimizer of g on [ t 1 , t 2 ] . By (102) and (103), we have

Since g is convex on [ a, b ] , the median of g on [ a, b ] is 0 , and [ t 1 , t 2 ] = { t ∈ [ a, b ] | g ( t ) ≤ 0 } , it is not hard to check that

Combining (100) and (104) we have

where the last inequality made use of C = L/ 10 .

The proof is complete by combining the discussions in (Case 1) and (Case 2).

C Comparison of Theorem 2.3 and Theorem 1 of [10]

We first restate Theorem 1 of [10] in the setting of fitting a single tree (note that [10] discussed random forest).

Proposition C.1 (Theorem 1 of [10]) Suppose Assumptions 2.2 and 2.1 hold true. Let ̂ f ( d ) ( · ) be the tree estimated by CART with depth d . Fixed constants α 2 > 1 , 0 < η < 1 / 8 , 0 < c < 1 / 4 and δ > 0 with 2 η < δ < 1 / 4 . Then there exists constant C > 0 such that for all n and d satisfying 1 ≤ d ≤ c log 2 ( n ) , it holds

In particular, the RHS of (105) is lower bounded by

Note that (106) follows (105) by the fact 1 ≤ d ≤ c log 2 ( n ) and α 2 > 1 . In the original Assumptions of Theorem 1 in [10], it was assumed that the noises can be heavy-tailed, which is a weaker assumption than Assumption 2.1. However, the parameter controlling the tails of the noises did not explicitly enter the error bound (105), and it seems that their proof techniques cannot improve the error bound even under the assumption that noises are bounded. In addition, the dependence on p was not explicitly stated in the bound (105), which seems to be hidden in the constant C .

To compare our error bound with the error bound in (106), since the ∥ ̂ f ( d ) -f ∗ ∥ 2 L 2 ( µ ) is bounded almost surely, it is not hard to transform the high-probability bound in (11) to an bound in expectation, and we have

Below we discuss two different cases.

(Case 1) λ ≥ 1 / 2 . Then it holds

So our convergence rate in (107) is O ( n -1 / 2 log( np ) log 2 ( n )) , but the rate in (106) is

(Case 2) 0 < λ ≤ 1 / 2 . Then it holds

and hence ϕ ( λ ) ≥ -log 2 (1 -λ ) / 2 . So our rate in (107) is O ( n log 2 (1 -λ ) / 2 log( np ) log 2 ( n )) , but the rate in (106) is

D Auxiliary results

Lemma D.1 (Bernstein's inequality) Let Z 1 , ...., Z n be i.i.d. random variables satisfying | E (( Z 1 -E ( Z 1 )) k ) | ≤ (1 / 2) k ! γ 2 b k -2 for some constants γ, b > 0 and for all k ≥ 2 . Then for any t > 0 ,

Lemma D.2 (Binomial tail bound) Let Z 1 , ..., Z n be i.i.d. random variables with P ( Z i = 1) = α and P ( Z i = 0) = 1 -α . Then for any t ∈ (0 , 1) ,

Lemma D.3 For any t ∈ (0 , 3 / 4) , log(1 -t ) > -t -t 2 .

Proof. For t ∈ (0 , 3 / 4) ,

Lemma D.4 Suppose Z is a random variable satisfying E ( e λZ ) ≤ e λ 2 σ 2 / 2 for all λ ∈ R , where σ > 0 is a constant; then

Proof. By Chernoff inequality it holds P ( | Z | > t ) ≤ 2 exp( -t 2 / (2 σ 2 )) for all t > 0 . As a result,

Lemma D.5 For any integer k ≥ 2 it holds 1 k 2 -4 ( k +1) 3 ≤ 1 ( k +1) 2 .

Proof. For any k ≥ 2 it holds

Multiplying 2 / ( k +1) 3 in the display above, we have

The proof is complete by noting that 2 k +1 k 2 ( k +1) 2 = 1 k 2 -1 ( k +1) 2 .

□

Lemma D.6 Let [ a, b ] be a sub-interval of [0 , 1] , and c ∈ ( a, b ) . Let h be a function on [ a, b ] such that h is differentiable on ( a, c ) and ( c, b ) , but can be discontinuous at c . Denote ∆ h ( c ) := lim t → c + h ( t ) -lim t → c -h ( t ) . Suppose

Then it holds

Proof. We assume that h is not continuous at c , since otherwise, the conclusion holds true trivially. We use the notation h ( c +) := lim t → c + h ( t ) and h ( c -) := lim t → c -h ( t ) . Without loss of generality, assume h ( c +) > h ( c -) .

For w ≥ (1 / 2)( h ( c +) + h ( c -)) , it holds w -h ( c -) ≥ (1 / 2)∆ h ( c ) . By (112), we know that for any t ∈ ( a, c ) ,

Hence for all t ∈ ( a, c ) ,

As a result,

For w < (1 / 2)( h ( c +) + h ( c -)) , similarly, we can prove

The proof is complete by combining (113) and (114).

□

Peer reviews:
comments: SUMMARY:
This paper improved the rate of consistency of CART based on a sufficient impurity decrease (SID) condition under regression settings. Then, the authors provided examples, which are mostly special additive models, that can satisfy the SID condition, and showed that the rate of consistency cannot be improved by more than a log factor.

STRENGTHS:
The strengths of this paper are summarized as follows:

- The "locally reverse Poincare inequality" shown in this paper is intuitive and can be more general than previous studies.
- The results are intuitive and improved the results in Chi et al. 2022 when the noise is sub-Gaussian.

WEAKNESSES:
The weaknesses of this paper are summarized as follows:

**About contributions**

1. The examples in this paper that satisfy the SID condition are still restricted to special additive models, which can be somehow not so practical. And it seems that extending the results to non-additive models is not easy. 
2. The discussions about optimality in [Line 279-285] are not so strict. Note that the lower bound of order $\Omega(n^{-2 / (p + 2)})$ shown in Tan et al. 2022 is based on general additive models which may not satisfy the SID condition. In other words, when the SID condition satisfies, the lower bound may be better.

Typos:
- $\kappa$ in [Line 242] seems to be not defined and it seems to be $\tau$.

QUESTIONS:
I would like to ask the authors the following questions:

- Can the SID condition be satisfied in multi-dimensional settings without assuming additive models? See Weaknesses #1.
- May the curse of dimensionality be avoided when the SID condition verifies? See Weaknesses #2.

LIMITATIONS:
The authors adequately addressed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the convergence rate of CART, a greedy algorithm for building decision trees, under a regression setting. It introduces a sufficient impurity decrease (SID) condition on the underlying function that ensures the consistency and polynomial convergence of CART. It also provides examples and sufficient conditions to verify the SID condition for various function classes, such as additive models, polynomials, and smooth and strongly convex functions.\n\nSTRENGTHS:\n1.\tThe paper provides a refined analysis of CART under the SID condition, which improves the prediction error bound over the previous work by Chi et al. (2022) and shows its optimality up to logarithmic factors.\n2.\tThe paper decodes the mystery of the SID condition by introducing a locally reverse Poincare (LRP) class of univariate functions and showing that additive functions with LRP components satisfy the SID condition. This connects the two types of assumptions in the literature: additive model and SID condition.\n3.\tThe paper demonstrates the practical utility of its results by providing examples and sufficient conditions to check the LRP property for various well-known function classes, such as polynomials, smooth and strongly convex functions, and strongly increasing functions.\n4.\tThe paper is well-written and clear, with detailed proofs and explanations in the appendix. It also includes simulations and figures to illustrate its main findings and compare them with existing results.\n\nWEAKNESSES:\n1.\tThe paper relies on some technical assumptions and conditions, such as bounded errors, bounded signal function, and LRP property, which may not hold or be easy to verify for some function classes. It would be helpful to provide more intuition and motivation for these assumptions and conditions, and discuss their implications and limitations for the applicability of the results.\n2.\tThe improvement of the convergence rate in this paper still heavily relies on the SID condition. Although the paper decomposes and illustrates the SID condition with some examples, they are still mathematical forms in 1-dimensional situations. If the paper could further analyze the SID condition in relation to the structure of real data, then its contribution to the SID condition would be more convincing.\n\nQUESTIONS:\n1.\tDo Examples 3.1 - 3.3 still hold if they are generalized to high-dimensional situations?\n2.\tThe additive model assumption in Section 3 is still quite strong, is it possible to use milder assumptions?\n3.\tThe intuition behind the LRP condition needs further explanation, does this mathematical property have any implications for improving the CART algorithm? Maybe it can help explain why CART tends to handle data with certain structural characteristics.\n\nLIMITATIONS:\nNAN.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
The paper performs theoretical analysis of the well-known CART algorithm. The authors show a convergence rate of CART under the condition called 'sufficient impurity decrease' (SID), which is tighter than known ones. The authors further provide the condition for a class of functions that satisfies SID.

STRENGTHS:
- The paper seems technically sound and the analysis of CART would be important. 

- The paper elaborately explains differences from past work in particular from [10].

WEAKNESSES:
- Experimental verification is only in one quite simple figure for the simplest case. Of-course, the paper is a theoretical paper, but if richer empirical evidences had been provided, it would have been convincing more (such as different true functions and comparison with past bounds).

QUESTIONS:
- Is there any insight that can be derived by the proposed analysis for tree ensemble models such as random forest?

LIMITATIONS:
A limitation about lambda is discussed in Section 5.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper focuses on the analysis of the prediction error of Classification and Regression Trees (CART) for regression problems under a sufficient impurity decrease (SID) condition. The SID condition is a strong assumption on the approximation power of tree splits, which can ensure the consistency of CART. The paper establishes an upper bound on the prediction error of CART under the SID condition and shows that the error bound is tight up to log factors. The paper also discusses a few sufficient conditions under which an additive model satisfies the SID condition. The paper's first contribution is a refined analysis of CART under the SID condition, which improves upon the known result by providing a tighter error bound. The paper's second contribution is the decoding of the mystery of the SID condition, which builds a connection between the two types of assumptions in the literature: the additive model and the SID condition. The paper discusses a few examples of how the locally reverse Poincare inequality can be verified. Overall, the paper provides a refined analysis of CART under the SID condition and sheds light on the connection between two types of assumptions in the literature: additive model and SID condition.\n\nSTRENGTHS:\nThe paper has several strengths. First, it provides a refined analysis of the prediction error of Classification and Regression Trees (CART) under a sufficient impurity decrease (SID) condition for regression problems. The paper establishes an upper bound on the prediction error of CART under the SID condition and shows that the error bound is tight up to log factors. Second, the paper discusses a few sufficient conditions under which an additive model satisfies the SID condition. Third, the paper provides examples of how the locally reverse Poincare inequality can be verified.\n\nWEAKNESSES:\nI think the work is significant. I don’t have many concerns, but I have some questions about the method.\n1. Can this method be extended to classification problems?\n2. In theorem 2.3, why do you need to suppose n satisfies such a complicated condition?\n3. Also, in theorem 2.3, why should d be fixed to a function of data size? \n4. In line 32, is it [0,1]^p?\n\nQUESTIONS:\nReferred to Strengths and Weaknesses\n\nLIMITATIONS:\nReferred to Strengths and Weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}

comments: SUMMARY:
In this paper, the approximation of the model obtained from the decision tree learning algorithm CART is analyzed. More precisely, in Theorem 2.3, a convergence rate of the decision tree approximation compared to the true model is obtained. This analysis is performed under two assumptions of the paper (already introduced in the literature), with the Sufficient Impurity Decrease (SID) assumption in particular. Finally, examples of true models that satisfy the SID assumption are proven.

STRENGTHS:
1. The paper is well-written and easy to follow (although I did not check the proofs).
2. The analysis of Theorem 2.3 improves the result (i.e., the convergence rate) of [10].
3. Proposition 3.1 links the SID condition and the class of functions satisfying the Locally Reverse Poincaré condition (that is proven for some classes).

WEAKNESSES:
1. The additive structure assumption of $f^*$ seems restrictive (but is also assumed in previous works).

QUESTIONS:
Typos and minor comments:
- l38: the the -> the
- l104: Chi et al.[10] -> Chi et al. [10]
- l144: the notations $\bar{y}\_{\mathcal{I}\_A}$, $\bar{y}\_{\mathcal{I}\_{A\_L}}$ and $\bar{y}\_{\mathcal{I}\_{A\_R}}$ are not introduced
- l163: "as" -> "at"
- l193: "Assumptions‘2.1" -> "Assumptions‘ 2.1"
- l240/246/247: Poincare -> Poincaré 
- l242: $\kappa$ -> "$\tau$"
- l704: there is an indefinite reference "??"
- The references are not "standardized", e.g. for JMLR it is written "The Journal of Machine Learning Research" or "Journal of Machine Learning Research"

LIMITATIONS:
The limitation about the the SID condition is discussed in Section 5. Moreover, I do not see any potential negative societal impact of their work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-20