SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yan Wang
Detroit, MI 48202
wangyan@wayne.edu
Huaiqing Wu, Dan Nettleton
{isuhwu,dnett}@iastate.edu
We establish stability of random forests under the mild condition that the squared response  $(Y^{2})$  does not have a heavy tail. In particular, our analysis holds for the practical version of random forests that is implemented in popular packages like randomForest in R. Empirical results show that stability may persist even beyond our assumption and hold for heavy-tailed  $Y^{2}$ . Using the stability property, we prove a non-asymptotic lower bound for the coverage probability of prediction intervals constructed from the out-of-bag error of random forests. With another mild condition that is typically satisfied when  $Y$  is continuous, we also establish a complementary upper bound, which can be similarly established for the jackknife prediction interval constructed from an arbitrary stable algorithm. We also discuss the asymptotic coverage probability under assumptions weaker than those considered in previous literature. Our work implies that random forests, with its stability property, is an effective machine learning method that can provide not only satisfactory point prediction but also justified interval prediction at almost no extra computational cost.
Random forests (RFs) is a successful machine learning method that serves as a standard approach to tabular data analysis and has good predictive performance [10, 7]. However, there is a big gap between the empirical effectiveness of RFs and the limited understanding of its properties. Most known theoretical results are established for variants of RFs not necessarily used in practice [5, 25, 22, 14, 30]. For the RF version implemented in packages like randomForest in R [21], little is known without strong assumptions [6, 26, 35]; RFs is notoriously difficult to analyze as a greedy algorithm. Here we show an important property for the RF used in practice (as well as for other variants) under realistic conditions.
1.1 Stability of random forests
The first main contribution of this work establishes the stability condition for the RF.
Theorem 1 (Stability of random forests, informal). For independent and identically distributed (iid) training data points  $(X_{i},Y_{i}), i\in \{1,\dots ,n\} \equiv [n]$  and a test point  $(X,Y)$ , if the squared response  $Y^{2}$  does not have a heavy tail, then the RF predictor  $\mathsf{RF}_B$  and any out-of-bag (OOB) predictor  $\mathsf{RF}_B^{\backslash i}$  predict similar values, i.e.,
$$
\mathbb {P} \left(\left| \mathsf {R F} _ {B} (X) - \mathsf {R F} _ {B} ^ {\backslash i} (X) \right| > \varepsilon_ {n, B}\right) \leqslant \nu_ {n, B}, \tag {1}
$$
where  $\mathsf{RF}_B$  results from the aggregation of all  $B$  base tree predictors, while  $\mathsf{RF}_B^{\backslash i}$  only those with the point  $(X_i,Y_i)$  excluded in training;  $\varepsilon_{n,B}$  and  $\nu_{n,B}$  are small numbers depending on  $n$  and  $B$ .
This result is referred to as the stability of the RF because it indicates that no single training point is extremely important in determining  $\mathsf{RF}_B$  in a probabilistic sense. Theorem 1 relies on a recent important work that establishes the absolute stability (see below for a precise definition) of general bagged algorithms with bounded outputs [28]. We take advantage of the fact that the range of the RF output is conditionally dependent upon the maximal and minimal values of  $Y$  in the training set, and then we show in theory that the stability property of the RF is possible even if  $Y$  is marginally unbounded. To our knowledge, this is the first stability result established for the RF.
The technique used in our analysis requires that  $Y^2$  not have a heavy tail (to make  $\varepsilon_{n,B}$  and  $\nu_{n,B}$  small). Though arguably already mild, we conjecture that this condition might be further relaxed. As shown below, numerical evidence suggests that the light-tail assumption may not be necessary for RF stability, which could hold even when  $Y$  follows a heavy-tail distribution like the Cauchy distribution.
1.2 Random-forest prediction intervals
Stability is a crucial property of a learning algorithm. For example, stability has a deep connection with the generalization error of an algorithm [9, 19, 23]. Moreover, stability also turns out to be important in distribution-free predictive inference. In particular, an algorithm being stable justifies the jackknife prediction interval (PI), which otherwise has no coverage guarantee [3].
In this work, we show that stability makes it possible to construct a PI with guaranteed coverage from the OOB error of the RF. The OOB error is defined as  $R_{i} = |Y_{i} - \mathsf{RF}_{B}^{\backslash i}(X_{i})|, i \in [n]$ . A main reason why such a PI is appealing is that  $R_{i}$  can be obtained almost without extra effort. For example, a one-shot training using the R package randomForest gives us an RF predictor  $\mathsf{RF}_B$  and all  $n$  OOB predictions  $\mathsf{RF}_B^{\backslash i}(X_i)$ . So, from the computational point of view, a convenient way to construct a PI for a test point  $(X,Y)$  is of the form "  $\mathsf{RF}_B(X) \pm$  proper quantile of  $\{R_i\}$ " [17, 35].
The second main contribution of this work constructs such PIs and theoretically proves, under mild conditions, the non-asymptotic lower and upper bounds for the coverage probability.
Theorem 2 (Coverage lower bound, informal). Under the same assumptions as in Theorem 1, and for  $\alpha \in (0,1)^1$ , we have the following lower bound of coverage probability:
$$
\mathbb {P} \left(\left| Y - \mathrm {R F} _ {B} (X) \right| \leqslant t h e \lceil (n + 1) (1 - \alpha) \rceil - t h s m a l l e s t R _ {i} + \varepsilon_ {n, B}\right) \geqslant 1 - \alpha - O (\sqrt {\nu_ {n , B}}),
$$
where  $\lceil \cdot \rceil$  is the ceiling function. Big  $O$  and other related notations are used in the usual way.
Theorem 3 (Coverage upper bound, informal). If we further assume that  $Y$  is continuous, resulting in distinct prediction errors, then we also have the following upper bound:
$$
\mathbb {P} \left(\left| Y - \mathrm {R F} _ {B} (X) \right| \leqslant t h e \lceil (n + 1) (1 - \alpha) \rceil - t h s m a l l e s t R _ {i} - \varepsilon_ {n, B}\right) \leqslant 1 - \alpha + \frac {1}{n + 1} + O (\sqrt {\nu_ {n , B}}).
$$
As we detail below, the PIs we provide coverage guarantees for are neither the jackknife-with-stability interval discussed in [3], nor the jackknife+-after-bootstrap interval established in [18]. In our context, constructing the former needs  $n$  leave-one-out (LOO) predictors (rather than  $n$  OOB predictors), i.e.,  $n$  additional RFs with each built on a training set of size  $n - 1$ . Constructing the latter needs the explicit information of each  $\mathsf{RF}_B^{\backslash i}(\cdot)$  rather than the OOB prediction  $\mathsf{RF}_B^{\backslash i}(X_i)$  for each  $X_i$  only. Both these methods require additional, sometimes extensive, computation given current popular packages. In contrast, our results are operationally more convenient. After one-shot training, we obtain not only a point predictor  $\mathsf{RF}_B(\cdot)$ , but also a valid interval predictor at almost no extra cost. Under reasonable conditions, our results indicate that by slightly inflating (or deflating) the PI constructed from the  $[(n + 1)(1 - \alpha)]$ -th smallest  $R_i$ , the coverage probability is guaranteed not to decrease (or increase) too much from the desired level of  $1 - \alpha$ . In fact, many numerical results, such as those in [17, 35], suggest that
$$
\mathbb {P} \left(\left| Y - \operatorname {R F} _ {B} (X) \right| \leqslant \text {t h e} [ (n + 1) (1 - \alpha) ] - \text {t h s m a l l e s t} R _ {i}\right) \approx 1 - \alpha .
$$
Motivated by this fact, we further establish an asymptotic result of coverage for such PIs.
<sup>1</sup>When  $\alpha \in (0, 1 / (n + 1))$ , we follow the convention that the  $(n + 1)$ -th smallest  $R_{i}$  is  $\infty$ .
Theorem 4. (Asymptotic coverage, informal) In addition to the conditions in the above theorems, also suppose the prediction error  $|Y - \mathsf{RF}_B(X)|$  is continuous, and its cumulative distribution function (CDF) does not change too drastically for all sufficiently large  $n$ . Then
$$
\mathbb {P} \left(\left| Y - \mathrm {R F} _ {B} (X) \right| \leqslant t h e \left[ (n + 1) (1 - \alpha) \right] - t h s m a l l e s t R _ {i}\right)\rightarrow 1 - \alpha a s n \rightarrow \infty .
$$
In [35], this asymptotic coverage was proved based on stronger assumptions. In particular, the true model is assumed to be additive such that  $Y = f_0(X) + \mathrm{noise}$  with the zero-mean noise independent of  $X$ , and  $\mathsf{RF}_B(X)$  is assumed to converge to  $f_0(X)$  in probability. We do not require  $\mathsf{RF}_B$  to converge to anything in any sense when  $n \to \infty$ . Technically, we need the family of prediction error CDFs be uniformly equicontinuous.
Based on our results, the RF seems to be the only one, among existing popular machine learning algorithms, that can provide both point and interval predictors with justification in such a convenient way. This makes the RF appealing, especially for tasks where the computational cost is a concern.
It is also worth noting that the upper-bound result is of interest in its own right. It can be generalized to jackknife PIs that are constructed from any stable algorithm; the result serves as a complement to the lower bounds established previously [3, 18].
Summarizing, we
- theoretically prove that the (greedy) RF algorithm is stable when  $Y^{2}$  does not have a heavy tail;
- numerically show that RF stability may hold beyond the above light-tail assumption;
- construct PIs based on the OOB error with finite-sample coverage guarantees: the lower bound of coverage does not need any additional assumption beyond stability; the upper bound needs an additional assumption, which is usually satisfied when  $Y$  is continuous;
- provide the upper bound of coverage for jackknife PIs constructed from general stable algorithms, assuming distinct LOO errors; and
- prove asymptotically exact coverage for RF-based PIs under weaker assumptions than those previously considered in published work.
2 Concepts of algorithmic stability
Stability stands at the core of this work. There are different types of stability, each of which is used to assess quantitatively how stable (in some certain sense) an algorithm is with respect to small variations in training data [9, 28, 4]. In a recent work [4], robust optimization is used to enhance the stability of algorithms in classification tasks. In [28], bagging is proved to be an efficient mechanism to stabilize algorithms in regression tasks. We focus on regression here. As will be made clear, the technique used in this work relies on the fact that the RF predictor in regression results from averaging tree predictors. However, the majority vote of tree predictors is used in classification, and new ideas are needed to analyze the RF stability in this setting. For our purposes, we introduce three levels of stability from strongest to weakest. The strongest version of stability, introduced in [28], does not depend on the data distribution, and may be referred to as "absolute stability."
Definition 1 (Absolute stability of algorithms). For any dataset consisting of  $n \geqslant 2$  training points  $D = \{(X_1, Y_1), \ldots, (X_n, Y_n)\}$  and any test point  $(X, Y)$ , an algorithm  $\mathcal{A}$  is defined to be  $(\varepsilon, \delta)$ -absolutely-stable if
$$
\frac {1}{n} \sum_ {i = 1} ^ {n} \mathbb {P} _ {\xi} \left(\left| \hat {f} (X) - \hat {f} ^ {- i} (X) \right| > \varepsilon\right) \leqslant \delta
$$
for some  $\varepsilon, \delta \geqslant 0$ , where  $\xi$  denotes the possible innate randomness in the algorithm (such as the node splitting procedure in the RF) and can be seen as a random variable uniformly distributed in  $[0,1]$ ,  $\hat{f} = \mathcal{A}(D;\xi)$  is the predictor trained on  $D$ , and  $\hat{f}^{-i} = \mathcal{A}(D^{-i};\xi)$  is the  $i$ th LOO predictor trained on  $D^{-i}$ , i.e.,  $D$  without the  $i$ th point  $(X_{i},Y_{i})$ . We might refer to the RF as both an algorithm (the learning procedure) and a predictor (the learned function) for simplicity.
Many bagged algorithms, in particular those with bounded predicted values, can achieve absolute stability with both  $\varepsilon$  and  $\delta$  converging to 0, as long as  $n$  and the number of bags  $B$  go to infinity.
However, the predicted value of the RF is in general unbounded (for regression tasks considered in this work), and we are more interested in another type of stability, investigated in [9], and called out-of-sample stability [3]. For simplicity, we name it "stability." This notion of stability turns out to be important in validating a jackknife prediction interval.
Definition 2 (Stability of algorithms). For iid training and test data, algorithm  $\mathcal{A}$  is  $(\varepsilon, \delta)$ -stable if
$$
\mathbb {P} _ {D, X, \xi} \left(\left| \hat {f} (X) - \hat {f} ^ {- i} (X) \right| > \varepsilon\right) \leqslant \delta
$$
for some  $\varepsilon, \delta \geqslant 0$ , where  $D, X, \hat{f}, \hat{f}^{-i}$  are as defined above.
We will establish this type of stability for the derandomized RF defined below, where the data-generating distribution is involved. To this end, we will use the methods in [28], which aim to provide absolute stability for bagged algorithms. Technically, we use such methods to first establish the "conditional stability" of an algorithm with respect to given data.
Definition 3 (Conditional stability of algorithms). Conditional on  $D$  and  $X$ , an algorithm  $\mathcal{A}$  is defined to be  $(\varepsilon, \delta)$ -conditionally-stable if
$$
\frac {1}{n} \sum_ {i = 1} ^ {n} \mathbb {P} _ {\xi | D, X} \left(\left| \hat {f} (X) - \hat {f} ^ {- i} (X) \right| > \varepsilon \Big | D, X\right) \leqslant \delta
$$
Once conditional stability is established for the derandomized RF algorithm, its stability can be consequently established by invoking
$$
\mathbb {P} _ {D, X, \xi} (\cdot) = \mathbb {E} _ {D, X} \left[ \mathbb {P} _ {\xi | D, X} (\cdot | D, X) \right].
$$
Stability of the derandomized RF provides the most essential ingredient for that of the practical RF, although the latter involves another type of stability, known as ensemble stability [18]. Ensemble stability justifies replacing the LOO predictor with the OOB predictor in (1). We may abuse the term "stability" in the following when the OOB, rather than the LOO, predictor is used.
3 Stability of random forests
3.1 Basics of random forests
This work mainly considers using the RF to perform regression tasks, where the response  $Y \in \mathbb{R}$  can be unbounded. By construction, the RF predictor with  $B$  bags, denoted by  $\mathrm{RF}_B$ , is a bagged algorithm with the base algorithm being a tree, and  $\mathrm{RF}_B = \frac{1}{B}\sum_{b=1}^{B}\mathrm{TREE}_b$ , where  $\mathrm{TREE}_b$  is the  $b$ th tree predictor, trained on the  $b$ th bag  $r_b$ , a bootstrapped sample of the training set  $D$ . The randomness in the tree predictor TREE originates from two independent sources: innate randomness  $\xi$  in the node splitting process and resampling randomness from the bag  $r$ . For the  $i$ th point, one can define the OOB RF predictor as  $\mathrm{RF}_B^{\backslash i} = \frac{1}{B_i}\sum_{b=1}^{B}\mathrm{TREE}_b \times \mathbb{I}\{i \notin r_b\}$ , where  $\mathbb{I}\{\cdot\}$  denotes the indicator function, and  $B_i = \sum_{b=1}^{B}\mathbb{I}\{i \notin r_b\}$ . Define  $p \equiv \mathbb{P}(i \in r)$  as the probability that the  $i$ th point is included in bag  $r$ . Then it is clear that  $B_i \sim \mathrm{Binomial}(B, 1 - p)$  for all  $i$ . We also denote  $r_f$  and  $r^{\backslash i}$  as the derandomized versions of  $\mathrm{RF}_B$  and  $\mathrm{RF}_B^{\backslash i}$ , respectively. Precisely,  $r_f = \mathbb{E}_{\xi,r}[\mathrm{TREE}]$  and  $r^{\backslash i} = \mathbb{E}_{\xi,r}[\mathrm{TREE}|i \notin r]$ . It is worth noting that, by definition,  $\mathrm{RF}_B^{\backslash i} \neq \mathrm{RF}_B^{-i}$  for finite  $B$ , while  $r^{\backslash i} = r^{\backslash i}$  as the derandomized RF results from the aggregation of an infinite number of trees. Since RF predictors are averages over tree predictors, the predicted values they output, given training set  $D$ , are bounded in  $[Y_{(1)}, Y_{(n)}]$ , where  $Y_{(1)}$  and  $Y_{(n)}$  are the minimum and maximum of  $\{Y_1, \ldots, Y_n\}$ , respectively. We also let  $Z_i = |Y_i|$  for all  $i$ , and denote the maximum as  $Z_{(n)}$ . As a result, we have that
$$
\left| \operatorname {r f} ^ {\backslash i} - \operatorname {r f} \right| \leqslant Y _ {(n)} - Y _ {(1)} \leqslant 2 Z _ {(n)}. \tag {2}
$$
Remark 1. This is also true for  $\mathsf{RF}_B^{\backslash i}$  and  $\mathsf{RF}_B$  for any finite  $B$ . In fact, this is a distinctive feature of the RF, irrespective of the node splitting rule. Other regression methods do not necessarily have such a data-dependence bound. This observation helps to establish the conditional stability of the RF.
Remark 2. Practically, when  $n$  is large, one might think that the bound (2) is crude. On one hand, if we look for a bound valid for any finite  $n \geqslant 2$ , then there is not much room for improvement for small  $n$ . On the other hand, we do expect that the typical stability of the RF can go beyond the finite-sample guarantee provided by (2) when  $n$  is big, which is consistent with the numerical results shown below. A more informative bound for large  $n$  is worth future investigation.
There are several quantities that are useful in establishing the RF stability; they can be calculated explicitly and are listed below. First, it is well known that
$$
p \equiv \mathbb {P} (i \in r) = 1 - (1 - 1 / n) ^ {n} = 1 - 1 / e + O (1 / n). \tag {3}
$$
Actually,  $p$  is monotonically decreasing for  $n \geqslant 1$ . Second,
$$
q \equiv - \operatorname {C o v} \left(\mathbb {I} \{i \in r \}, \mathbb {I} \{j \in r \}\right) = (1 - 1 / n) ^ {2 n} - (1 - 2 / n) ^ {n} = O (1 / n), \tag {4}
$$
as can be directly checked. Third, the moment generating function of  $B_{i}$  is
$$
\mathbb {E} \left[ e ^ {s B _ {i}} \right] = (p + (1 - p) e ^ {s}) ^ {B}. \tag {5}
$$
In the following, we first perform the stability analysis for the derandomized RF (consisting of an infinite number of trees) and then extend the results to the practical finite- $B$  case.
3.2 Derandomized random forests
The following theorem formalizes the conditional stability property for the derandomized RF, the proof of which is a direct result of Theorem 8 in [28], and is omitted here.
Theorem 5 (Conditional stability of derandomized random forests). Conditional on training set  $D$  and test point  $(X, Y)$ , for the derandomized random forest predictor  $r$  we have that
$$
\frac {1}{n} \sum_ {i = 1} ^ {n} \mathbb {I} \left\{\left| \operatorname {r f} (X) - \operatorname {r f} ^ {\backslash i} (X) \right| > \varepsilon \mid D, X \right\} \leqslant \delta (D, X) \equiv \frac {Z _ {(n)} ^ {2}}{\varepsilon^ {2} n} \left(\frac {p}{1 - p} + \frac {q}{(1 - p) ^ {2}}\right). \tag {6}
$$
If  $\delta(D, X) \geqslant 1$ , the statement is trivial, and we will focus on the case that  $\delta(D, X) \in (0, 1)$ . We can now establish the stability property for the derandomized RF.
Theorem 6 (Stability of derandomized random forests). For iid training and test data and  $\varepsilon >0$ , the derandomized random forest predictor rf is stable with
$$
\mathbb {P} _ {D, X} \left(\left| \operatorname {r f} (X) - \operatorname {r f} ^ {\backslash i} (X) \right| > \varepsilon\right) \leqslant \frac {\mathbb {E} \left[ Z _ {(n)} ^ {2} \right]}{\varepsilon^ {2} n} \left(\frac {p}{1 - p} + \frac {q}{(1 - p) ^ {2}}\right) \equiv \nu . \tag {7}
$$
This result follows directly from the conditional stability (6) by averaging over  $D$  and  $X$ . There is some freedom in choosing the dependence of  $\varepsilon$  on  $n$ . On one hand, in order to make sense of the word "stability," we do expect  $\varepsilon$  and  $\nu$  to be small for large  $n$ . From (3) and (4), it is clear that the asymptotic behavior of  $\nu$  is dominated by that of  $\mathbb{E}[Z_{(n)}^2] / (\varepsilon^2 n)$ , which can be tuned by manipulating  $\varepsilon$ . For example, a matching convergence rate to 0 between  $\varepsilon$  and  $\nu$  might be desirable, and one can then set  $\varepsilon = O((\mathbb{E}[Z_{(n)}^2] / n)^{1/3})$  if the scaling of  $\mathbb{E}[Z_{(n)}^2] = o(n)$  is known or can be inferred. On the other hand, we can fix  $\varepsilon$  to further investigate the relation between stability and the convergence-in-probability property of the RF. By (7), under the condition that  $\mathbb{E}[Z_{(n)}^2] / n \to 0$  as  $n \to \infty$ , one immediately comes to the conclusion that  $\mathrm{rf}^{\backslash i}(X) - \mathrm{rf}(X)$  converges to 0 in probability. Actually, a stronger conclusion can be drawn under the same condition.
Corollary 1. For iid training and test data, we have
$$
\mathbb {E} _ {D, X} \left[ | \operatorname {r f} (X) - \operatorname {r f} ^ {\backslash i} (X) | \right] <   2 \sqrt {\frac {\mathbb {E} \left[ Z _ {(n)} ^ {2} \right]}{n} \left(\frac {p}{1 - p} + \frac {q}{(1 - p) ^ {2}}\right)}. \tag {8}
$$
Further assume that  $\mathbb{E}[Y^2] < \infty$ . Then we have
$$
\mathbb {E} _ {D, X} \left[ | \operatorname {r f} (X) - \operatorname {r f} ^ {\backslash i} (X) | \right]\rightarrow 0 a n d \operatorname {r f} (X) - \operatorname {r f} ^ {\backslash i} (X) \xrightarrow {\mathbb {P}} 0 a s n \rightarrow \infty . \tag {9}
$$
Remark 3. The additional assumption that  $\mathbb{E}[Y^2] < \infty$  is mild. Many commonly encountered random variables have a light tail and thus a finite second moment, irrespective of the detailed information of the distribution in question. Note that the bound (2) itself can be crude, and our result is expected to be valid even beyond this mild condition.
Remark 4. This result indicates that the difference is diminishing between  $\mathsf{rf}$  and  $\mathsf{rf}^{\backslash i}$ , built on  $n$  and  $n - 1$  training data points, respectively. However, there is no indication that the derandomized  $\mathsf{rf}(X)$  itself will converge to anything. This idea inspires the proposal of Theorem 11.
The proof of this result, as well as others below, will be deferred to the Appendix. So far, we have investigated the derandomized version of the RF, which is a limiting case and can be seen as consisting of an infinite number of trees, averaging out all kinds of possible randomness in the predictor construction process. In order to make the results more relevant to applied machine learning, the finite- $B$  analysis for the RF is conducted below.
3.3 Finite- $B$  random forests
We now consider the difference between  $\mathsf{RF}_B$  and  $\mathsf{RF}_B^{\backslash i}$ . We denote  $\pmb {\xi} = (\xi_{1},\dots ,\xi_{B})$  and  $\boldsymbol {r} = (r_1,\ldots ,r_B)$  as the corresponding sources of randomness in  $B$  trees. We also consider conditional stability first and then move to the stability of  $\mathsf{RF}_B$ .
Theorem 7 (Conditional stability of finite- $B$  random forests). Conditional on training set  $D$  and test point  $(X,Y)$ , for a random forest predictor  $\mathsf{RF}_B$  that consists of  $B$  trees, we have for  $\varepsilon >0$  that
$$
\frac {1}{n} \sum_ {i = 1} ^ {n} \mathbb {P} _ {\pmb {\xi}, \pmb {r} | D, X} \left(\left| \mathsf {R F} _ {B} (X) - \mathsf {R F} _ {B} ^ {\backslash i} (X) \right| > \varepsilon + 2 \sqrt {\frac {2 Z _ {(n)} ^ {2}}{B} \ln \left(\frac {1}{\delta}\right)} \Bigg | D, X\right) \leqslant 3 \delta + g (p, \delta , B),
$$
where  $\delta$  is short for  $\delta (D,X)$  as defined in (6) and  $g(p,\delta ,B) = 2(p + (1 - p)\delta^{\frac{1}{B}})^{B}$ .