SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Rahul Mazumder *
Sloan School of Management  
Massachusetts Institute of Technology  
rahulmaz@mit.edu
Haoyue Wang
Operations Research Center  
Massachusetts Institute of Technology  
haoyuew@mit.edu
The decision tree is a flexible machine learning model that finds its success in numerous applications. It is usually fitted in a recursively greedy manner using CART. In this paper, we investigate the convergence rate of CART under a regression setting. First, we establish an upper bound on the prediction error of CART under a sufficient impurity decrease (SID) condition [10] – our result improves upon the known result by [10] under a similar assumption. Furthermore, we provide examples that demonstrate the error bound cannot be further improved by more than a constant or a logarithmic factor. Second, we introduce a set of easily verifiable sufficient conditions for the SID condition. Specifically, we demonstrate that the SID condition can be satisfied in the case of an additive model, provided that the component functions adhere to a "locally reverse Poincaré inequality". We discuss several well-known function classes in non-parametric estimation to illustrate the practical utility of this concept.
The decision tree [5] is one of the most fundamental and popular methods in the machine learning toolbox. It utilizes a flowchart-like structure to recursively partition the data and allows users to derive interpretable decisions. It is usually constructed in a data-dependent greedy manner, with an algorithm called CART [5]. Thanks to its computational efficiency and ability to capture nonlinear structures, decision trees have served as the foundation for various influential algorithms for ensemble learning, including bagging [6], random forest [7] and gradient boosting [11]. These algorithms are among the best-known nonparametric models for supervised learning with tabular data [14].
Despite its remarkable empirical success in various applications, our theoretical understanding of decision trees remains somewhat limited. The data-dependent splits employed in CART pose challenges for rigorous theoretical analysis. While some easy-to-analyze variants of CART have been widely studied [18, 4, 3], rigorous theoretical analysis of the original version of CART has been absent until recently. In particular, a recent line of work [19, 24, 10, 16] has established the consistency and prediction error bounds for CART and random forest. See Section 1.1 for a comprehensive review of related literature.
In this paper, we study the decision tree under a nonparametric regression model:
$$
y _ {i} = f ^ {*} \left(x _ {i}\right) + \epsilon_ {i} \quad i \in [ n ] \tag {1}
$$
with  $x_{i} = (x_{i}^{(1)},\dots,x_{i}^{(p)}) \in \mathbb{R}^{p}, y_{i} \in \mathbb{R}; \epsilon_{i}$  is the noise with zero-mean, and  $f^{*}(x)$  is the signal function. We consider a random design where  $\{x_{i}\}_{i=1}^{n}$  are i.i.d. from some underlying distribution  $\mu$  supported on  $[0,1]^{p}$ , and  $\{\epsilon_{i}\}_{i=1}^{n}$  are i.i.d. independent of  $\{x_{i}\}_{i=1}^{n}$ . For an estimator  $\hat{f}$  of  $f^{*}$ , we use
*This work is supported by Office of Naval Research (N00014-21-1-2841).
the  $L^2$  prediction error  $\| \hat{f} - f^* \|_{L^2(\mu)}^2$  as a measure of the estimator  $\hat{f}$ . Consider a regression tree algorithm that produces a heuristic (not necessarily exact) solution  $\hat{f}$  of
$$
\min  _ {f \in \mathcal {T} _ {d}} \sum_ {i = 1} ^ {n} \left(f \left(x _ {i}\right) - y _ {i}\right) ^ {2} \tag {2}
$$
where  $\mathcal{T}_d$  is the space of binary (i.e. each splitting node has two children) axis-aligned regression trees with maximum depth  $d$ . The statistical performance of the estimator  $\hat{f}$ , therefore, is affected by the following three factors: (i) Approximation error: how accurately can  $f^*$  be approximated by a regression tree in  $\mathcal{T}_d$ . (ii) Estimation error: the variance of the estimator, which is affected by the complexity of the function space  $\mathcal{T}_d$ . A larger space  $\mathcal{T}_d$  (i.e. larger  $d$ ) can reduce the approximation error, but it may also increase the estimation error and lead to overfitting. (iii) Optimization error: the discrepancy between  $\hat{f}$  and the exact optimal solution of the optimization problem (2). Indeed, if  $\hat{f}$  is the exact optimal solution of (2) (i.e. optimal regression tree), it can approximate a broad class of  $f^*$  [9], and the prediction error can be analyzed via the classical theory of least-squares estimators [22, 13]. However, since solving (2) to optimality is computationally challenging for large-scale data, heuristics like CART are typically employed in practice. To understand the statistical performance of CART, the major difficulty lies in the analysis of the optimization error of CART.
As a greedy algorithm, CART examines only one split in each iteration (see Section 1.3 for details). Although decision trees are intended to capture multivariate structures across different features, the greedy nature of CART poses challenges in approximating functions  $f^{*}$  that have complex multivariate structures. For example, consider the two-dimensional "XOR gate" function  $f^{*}(u) \coloneqq 1_{\{u \in [0,1/2)^2\}} + 1_{\{u \in [1/2,1)^2\}}$  [14], and assume that  $x_i$  has a uniform distribution on  $[0,1]^2$ . Note that  $f^{*}$  itself is a depth-2 decision tree, with (e.g.) the root node split with the first coordinate at  $1/2$ , and the left and right children split with the second coordinate at  $1/2$ . However, due to the symmetry structure of  $f^{*}$ , when the sample size  $n$  is large, the root split of CART is completely fitting to noises and is likely to generate a split near the boundary [5, 8]. Consequently, the resulting estimator  $\hat{f}$  fails to capture the true tree structure encoded in  $f^{*}$ . The major difficulty in fitting the XOR gate function with CART lies in the complicated interaction between the two coordinates.
The discussions above naturally lead to the following two questions about CART:
- What conditions on  $f^{*}$  are needed such that CART can approximate  $f^{*}$  and yield a consistent estimator?
- If  $f^{*}$  satisfies such conditions, what is the convergence rate of the prediction error of CART?
In this paper, we aim to address these fundamental questions by providing sufficient conditions on  $f^*$  such that one can establish convergence rates of CART. These sufficient conditions are quite flexible to include a broad class of nonparametric models. In the following, we first discuss some related literature and the basics of CART. In Section 2, we prove a general error bound for CART under a sufficient impurity decrease (SID) condition [10] of  $f^*$ . This error bound is an improvement over the existing results under similar conditions [10]. In Section 3, we introduce a sufficient condition for the SID condition on  $f^*$ , and use it to show that the SID condition can be satisfied by a broad class of  $f^*$ .
1.1 Related literature
The inception and analysis of decision trees and CART algorithm can be traced back to the seminal work of Breiman [5]. Breiman's study marked the first significant step towards establishing the consistency of CART, albeit under stringent assumptions concerning tree construction. The original formulation of CART, characterized by data-dependent splits, poses challenges when it comes to rigorous mathematical analysis. To address this issue, subsequent research efforts [18, 4, 3] have introduced alternative variants of CART that lend themselves to more tractable theoretical examinations. For instance, certain studies [12, 4, 3, 2] assume random selection of splitting features in the tree. Other approaches incorporate random selections of splitting thresholds [12, 2], while some restrict splits to occur exclusively at the midpoint of the corresponding cell in the chosen coordinate [3]. These simplifications ensure the independence between the splitting rules of the tree and the training data – the tree depends on the data only via the estimation in each cell. Such independence simplifies the analysis of the resultant trees. In a similar vein, alternative strategies have
been proposed, such as the employment of a two-sample approach [23] to compute splitting rules and cell estimations separately, giving rise to what is known as an "honest tree". This methodology also ensures independence between various aspects of the tree construction and estimation process.
Since the first consistency result in [5], the strict theoretical analysis of the original CART algorithm has remained elusive until recent advancements. A significant breakthrough came with the seminal work of Scornet et al. [19], which demonstrated the consistency of CART under the assumption of an additive model [15] with continuous component functions. It makes a technical contribution on the decrease of optimization error of CART. This technical argument strongly relies on the additive model. Subsequent works [17, 16] further extended the theoretical analysis by providing non-asymptotic error bounds for CART under a similar additive model. However, the rates derived in [17, 16] is a slow rate of  $O((\log (n))^{-1})$ , which seems not easy to be improved under the same assumption [8]. Although the additive model is a well-studied model in statistical learning, it is not immediately evident why such a modeling assumption is specifically required for CART to function effectively.
Another line of work [24, 10] adopts more intuitive assumptions on the signal function  $f^{*}$  to ensure the progress of CART. In particular, Chi et al. [10] introduced a sufficient impurity decrease (SID) condition. In essence, the SID condition posits that for every rectangular region  $B$  within the feature space, there exists an axis-aligned split capable of reducing the population variance of  $f^{*}$  (conditional on  $B$ ) by a fixed ratio. Note that such a split cannot be directly identified by CART, as it necessitates knowledge of the population distribution of  $x_{i}$ . Instead, CART employs empirical variance of  $y_{i_{i=1}}^{n}$  to guide its cell splitting decisions. Nonetheless, the SID condition is a strong assumption on the approximation power of tree splits, which can ensure the consistency of CART via an empirical process argument [10]. A similar condition has also appeared implicitly in [24]. Under the SID condition, Chi et al. [10] studied the consistency and convergence rates of CART, demonstrating a polynomial convergence  $O(n^{-\gamma})$  of the prediction error. A possible limitation of the SID condition is the difficulty of verification: it necessitates the decreasing condition to hold true for all rectangles. Although a few examples are provided in [10], the general procedure for verifying whether a specific  $f^{*}$  satisfies this condition remains unclear. Notably, its relationship with the additive model assumption in [19, 17] remains to be fully understood.
Lastly, it is worth mentioning that several recent studies [20, 8] have investigated the performance lower bounds of CART, shedding light on its limitations. Specifically, Tan et al. [20] demonstrated that even when assuming an additive regression function  $f^{*}$ , any single-tree estimator still suffers from the curse of dimensionality. Consequently, these estimators can only achieve convergence rates considerably slower than the oracle rates established for additive models [21]. Another study by Cattaneo et al. [8] delved into the splitting behavior of CART and provided theoretical support for empirical observations [5] indicating that CART tends to generate splits near boundaries in cases where the signal is weak. These lower bounds support the importance of using trees in ensemble models to reduce the prediction error. Although we focus on analyzing CART for a single tree in this paper, the result can be adapted to the analysis of ensembles of trees for the fitting of more complex function classes.
The first contribution of our paper is a refined analysis of CART under the SID condition. Although the convergence of CART has been studied in [10] under SID condition, the analysis of [10] seems not tight when the noises have light-tails (see discussions in the Appendix C). We establish an improved analysis of CART under this setting, and show by examples that our error bound is tight up to log factors.
The second contribution of this paper is the decoding of the mystery of the SID condition. We discuss a sufficient condition under which the SID condition holds true. In particular, we introduce a class of univariate functions that satisfy an locally reverse Poincaré inequality, and show that additive functions with each univariate component being in this class satisfy the SID condition. This builds a connection between the two types of assumptions in the literature: additive model and SID condition. We discuss a few examples that how the locally reverse Poincaré inequality can be verified.
1.3 Basics of CART
We review the methodology of CART to build the tree. The primary objective of decision trees is to find optimal partitions of feature space that produce a minimal variance of response variables. CART constructs the tree and minimizes the variance using a top-down greedy approach. It starts with a root node that represents the whole feature space  $\mathbb{R}^p$  – this can be viewed as an initial tree with depth 0. At each iteration, CART splits all the leave nodes in the current tree into a left child and a right child; the depth of the tree increases by 1, and the number of leave nodes in the tree doubles. At each leave node, it takes an axis-aligned splits with maximum variance (impurity) decrease. Let  $\mathcal{E}$  be the set of all intervals (which can be open, closed, or open on one side and closed on another side). Define the set of rectangles in  $[0,1]^p$  as:
$$
\mathcal {A} := \left\{\prod_ {j = 1} ^ {p} E _ {j} \mid E _ {j} \in \mathcal {E} \forall j \in [ p ] \right\} \tag {3}
$$
For each  $A \in \mathcal{A}$ , define  $\mathcal{I}_A \coloneqq \{i \in [n] \mid x_i \in A\}$ . For a leave node that represents a rectangle  $A \in \mathcal{A}$ , the impurity decrease of a split in feature  $j \in [p]$  with threshold  $b \in \mathbb{R}$  is given by:
$$
\widehat {\Delta} (A, j, b) := \frac {1}{n} \sum_ {i \in \mathcal {I} _ {A}} \left(y _ {i} - \bar {y} _ {\mathcal {I} _ {A}}\right) ^ {2} - \frac {1}{n} \sum_ {i \in \mathcal {I} _ {A _ {L}}} \left(y _ {i} - \bar {y} _ {\mathcal {I} _ {A _ {L}}}\right) ^ {2} - \frac {1}{n} \sum_ {i \in \mathcal {I} _ {A _ {R}}} \left(y _ {i} - \bar {y} _ {\mathcal {I} _ {A _ {R}}}\right) ^ {2} \tag {4}
$$
where  $y_{\mathcal{I}} \coloneqq \frac{1}{|\mathcal{I}|} \sum_{i \in \mathcal{I}} y_i$  for any  $\mathcal{I} \subseteq [n]$ , and
$$
A _ {L} = A _ {L} (j, b) := A \cap \{v \in [ 0, 1 ] ^ {p} \mid v _ {j} \leq b \},
$$
$$
A _ {R} = A _ {R} (j, b) := A \cap \{v \in [ 0, 1 ] ^ {p} \mid v _ {j} > b \}. \tag {5}
$$
To split on  $A$ , CART takes  $\hat{j}$  and  $\hat{b}$  that maximize the impurity decrease, i.e.,
$$
\hat {j}, \hat {b} \in \operatorname * {a r g m a x} _ {j \in [ p ], b \in [ 0, 1 ]} \widehat {\Delta} (A, j, b) \tag {6}
$$
This splitting procedure is repeated until the tree has grown to a given maximum depth  $d$ .
1.4 Notations
For  $a, b > 0$ , we write  $a = O(b)$  if there exists a universal constant  $C > 0$  such that  $a \leq Cb$ ; we write  $b = \Omega(a)$  if  $a = O(b)$ , and write  $a = \Theta(b)$  if both  $a = O(b)$  and  $a = \Omega(b)$  are true. For values  $a_{n,p}, b_{n,p}$  that may depend on  $n$  and  $p$ , we write  $a_{n,p} = \widetilde{O}(b_{n,p})$  if  $a_{n,p} = O(b_{n,p} \log^{\gamma}(np))$  for some fixed constant  $\gamma \geq 0$ .
2 Error bound of CART under SID property
We study the prediction error bound of CART for the regression problem (1). We focus on a random design as in the following assumption.
Assumption 2.1 (i) (Random design) Suppose  $\{x_{i}\}_{i = 1}^{n}$  are i.i.d. random variables with a distribution  $\mu$  supported on  $[0,1]^p$ . Suppose  $\mu$  has a density  $p_X$  (w.r.t. Lebesgue measure) on  $[0,1]^p$  satisfying  $0 < \underline{\theta} \leq p_X(u) \leq \bar{\theta} < \infty$  for all  $u \in [0,1]^p$  for some constants  $\underline{\theta} \leq 1$  and  $\bar{\theta} \geq 1$ .
(ii) (Error distribution) Suppose  $\{\epsilon_i\}_{i=1}^n$  are i.i.d. zero-mean bounded random variables with  $|\epsilon_i| \leq m < \infty$  for some  $m > 0$ . Suppose  $\{\epsilon_i\}_{i=1}^n$  are independent to  $\{x_i\}_{i=1}^n$ .
(iii) (Bounded signal function) Suppose  $\sup_{u\in [0,1]^p}|f^* (u)|\leq M < \infty$  for some constant  $M$
In Assumption 2.1 (i), it is assumed that the features  $x_{i}$  are supported on  $[0,1]^p$  for convenience. The same results (different by at most a constant) can be proved if we replace  $[0,1]^p$  with an arbitrary bounded rectangle. The second assumption (Assumption 2.1 (ii)) requires the error terms  $\{\epsilon_i\}_{i = 1}^n$  to be i.i.d. bounded - this is the key difference of our paper and the assumptions made in [10]. In particular, [10] made a milder assumption on the error terms that allows them to have heavy tails. Their result, however, when specifying to the case of bounded noises, is not tight (see
discussions in Appendix C for a comparison). Note that for convenience we have assumed that  $\epsilon_{i}$  is bounded. If instead, we assume the noises are sub-Gaussian, similar conclusions can also be proved via a truncation argument (the bound may be different by a log factor). The third assumption (Assumption 2.1 (iii)) is a standard assumption for nonparametric regression models [22, 13].
Under the random design setting in Assumption 2.1, we are ready to introduce the sufficient impurity decrease condition on the underlying function  $f^{*}$ . Let  $X$  be a random variable in  $\mathbb{R}^p$  that has the same distribution as  $x_{1}$  and is independent to  $\{x_{1},\ldots ,x_{n}\}$ . For any  $A\in \mathcal{A}$ ,  $j\in [p]$  and  $b\in \mathbb{R}$ , define:
$$
\begin{array}{l} \Delta (A, j, b) := \mathbb {P} (X \in A) \operatorname {V a r} \left(f ^ {*} (X) | X \in A\right) - \mathbb {P} (X \in A _ {L}) \operatorname {V a r} \left(f ^ {*} (X) | X \in A _ {L}\right) \\ - \mathbb {P} (X \in A _ {R}) \operatorname {V a r} \left(f ^ {*} (X) \mid X \in A _ {R}\right) \\ \end{array}
$$
where  $A_{L}$  and  $A_{R}$  are defined in (5), and  $\operatorname{Var}(f^{*}(X)|X\in A)$  means the conditional variance defined as
$$
\operatorname {V a r} \left(f ^ {*} (X) | X \in A\right) := \mathbb {E} \left(\left(f ^ {*} (X) - \mathbb {E} \left(f ^ {*} (X) | X \in A\right)\right) ^ {2} \mid X \in A\right) \tag {8}
$$
It is not hard to see the similarity between  $\Delta(A, j, b)$  and  $\widehat{\Delta}(A, j, b)$  (defined in (4)), where the former can be viewed as a population variant of the latter. Intuitively, for a split with feature  $j$  and threshold  $b$ ,  $\Delta(A, j, b)$  measures variance decrease of  $f^{*}$  after this split. The SID condition below assumes that there is a split with a large variance decrease.
Assumption 2.2 (Sufficient Impurity Decrease) There exists a constant  $\lambda \in (0,1]$  such that for all  $A\in \mathcal{A}$
$$
\sup  _ {j \in [ p ], b \in \mathbb {R}} \Delta (A, j, b) \geq \lambda \cdot \mathbb {P} (X \in A) \operatorname {V a r} \left(f ^ {*} (X) \mid X \in A\right) \tag {9}
$$
Note that Assumption 2.2 is a condition depending on both the function  $f^{*}$  and the underlying distribution  $X \sim \mu$ . Briefly speaking, the SID condition requires that the best split must decrease the population variance by a constant factor (note that  $\lambda$  is required to be bounded away from 0). Intuitively, the condition (9) can be satisfied if  $f^{*}$  has a significant "trend" along some axis in  $A$ , but may be hard to be satisfied if  $f^{*}$  is relatively "flat" in  $A$ . Since Assumption 2.2 requires (9) being satisfied for all cells  $A \in \mathcal{A}$ , it essentially requires that  $f^{*}$  is not "too flat" on any local rectangle  $A$ . In Section 3, we develop a technical argument to translate this intuition into a rigorous statement, and use it to check Assumption 2.2 for a wide class of functions.
With Assumptions 2.1 and 2.2 at hand, we are ready to present the main result in this section. For any function  $g: \mathbb{R}^p \to \mathbb{R}$ , let  $\| g \|_{L^2(\mu)}$  denote the  $L^2$ -norm of  $g$  with respect to the measure  $\mu$ .
Theorem 2.3 Suppose Assumptions 2.1 and 2.2 hold true. Let  $\widehat{f}^{(d)}(\cdot)$  be the tree estimated by CART with depth  $d$ . Suppose  $n$  is large enough such that  $\frac{2\bar{\theta}e^2d}{n}\vee \frac{\log(72p^d(n + 1)^{2d} / \delta)}{n} < 3 / 4$ . Then there exists a universal constant  $C > 0$  such that with probability at least  $1 - \delta$ , it holds that for any  $\alpha >0$
$$
\left\| \widehat {f} ^ {(d)} - f ^ {*} \right\| _ {L ^ {2} (X)} ^ {2} \leq 2 \operatorname {V a r} \left(f ^ {*} (X)\right) \cdot \left(1 - \frac {\lambda}{(1 + \alpha) ^ {2}}\right) ^ {d} + C \frac {2 ^ {d} (d \log (n p) + \log (1 / \delta))}{\alpha n} U ^ {2} \tag {10}
$$
where  $U\coloneqq M + m$  . In particular, taking  $\alpha = 1 / d$  and  $d = \lceil \log_2(n) / (1 - \log_2(1 - \lambda))\rceil$  , it holds
$$
\left\| \widehat {f} ^ {(d)} - f ^ {*} \right\| _ {L ^ {2} (X)} ^ {2} \leq C _ {\lambda , U} \frac {\log^ {2} (n) \log (n p) + \log (n) \log (1 / \delta)}{n ^ {\phi (\lambda)}} \tag {11}
$$
where  $\phi (\lambda)\coloneqq \frac{-\log_2(1 - \lambda)}{1 - \log_2(1 - \lambda)}$ , and  $C_{\lambda ,U}$  is a constant that only depends on  $\lambda$  and  $U$ .
The error bound in (10) consists of two terms. The first term corresponds to the bias of the approximating  $f^{*}$  using CART. It decreases geometrically as  $d$  increases, which is suggested by the SID condition. The second term is  $O(2^{d}d\log (np) / n)$ , which corresponds to the estimation variance of CART. Here the term  $2^{d}$  represents the model complexity - a fully-grown depth  $d$  tree has  $2^{d}$  leaves. The proof of Theorem 2.3 is presented in Appendix A. It is based on a careful technical argument that controls the difference between the population impurity decrease  $\Delta (A,j,b)$  and the empirical impurity decrease  $\widehat{\Delta} (A,j,b)$ . In particular, our analysis is different from that of [10]. Note that [10] makes a different assumption on the error distribution, and their technical argument is not sufficient for the proof of Theorem 2.3. Particularly, the error bound in (11) is an improvement of the results proved in Theorem 1 of [10]—see Section C for a detailed comparison.
Note that the convergence rate in (11) has a crucial dependence on the coefficient  $\lambda$ . For a large value of  $\lambda$ , the exponent  $\phi(\lambda)$  is also larger, leading to a faster convergence rate in (11). On the other end, if  $\lambda$  is small, as  $\lambda \to 0$ , we have  $\phi(\lambda) \to 0$ , and the convergence rate in (11) can be arbitrarily slow. Nonetheless, as long as the SID condition is satisfied with  $\lambda$  bounded away from 0, (11) shows a polynomial rate, which is better than the logarithmic rate in [17, 16] without the SID condition.
To explore the tightness of the error bounds in Theorem 2.3, we consider a simple example with  $f^{*}$  being a univariate linear function.
Example 2.1 (Univariate linear function) Suppose  $p = 1$ , and  $x_{i}$  are i.i.d. from the uniform distribution on [0, 1]. Suppose  $f^{*}$  is a univariate linear function:  $f^{*}(t) = t$ . Then the SID condition (Assumption 2.2) is satisfied with coefficient  $\lambda = 3/4$ . As a result, it holds  $\phi(\lambda) = 2/3$ , and the error bound in the RHS of (11) is  $\widetilde{O}(n^{-2/3})$ .
Proof. For any interval  $A = [\ell, r]$ , we have
$$
\mathbb {P} (X \in A) \operatorname {V a r} (f ^ {*} (X) | X \in A) = \int_ {\ell} ^ {r} \left(t - \frac {\ell + r}{2}\right) ^ {2} \mathrm {d} t = \frac {(r - \ell) ^ {3}}{1 2} \tag {12}
$$
Take  $b\coloneqq (\ell +r) / 2$  , then we have