Title: Stability of Random Forests and Coverage of Random-Forest Prediction Intervals
Venue: NeurIPS 2023
Accepted: yes
Keywords: Stability

Prediction Intervals

Random Forests
Abstract: We establish stability of random forests under the mild condition that the squared response ($Y^2$) does not have a heavy tail. In particular, our analysis holds for the practical version of random forests that is implemented in popular packages like \texttt{randomForest} in \texttt{R}. Empirical results show that stability may persist even beyond our assumption and hold for heavy-tailed $Y^2$. Using the stability property, we prove a non-asymptotic lower bound for the coverage probability of prediction intervals constructed from the out-of-bag error of random forests. With another mild condition that is typically satisfied when $Y$ is continuous, we also establish a complementary upper bound, which can be similarly established for the jackknife prediction interval constructed from an arbitrary stable algorithm. We also discuss the asymptotic coverage probability under assumptions weaker than those considered in previous literature. Our work implies that random forests, with its stability property, is an effective machine learning method that can provide not only satisfactory point prediction but also justified interval prediction at almost no extra computational cost.

Stability of Random Forests and Coverage of Random-Forest Prediction Intervals

Abstract

We establish stability of random forests under the mild condition that the squared response ( Y 2 ) does not have a heavy tail. In particular, our analysis holds for the practical version of random forests that is implemented in popular packages like randomForest in R . Empirical results show that stability may persist even beyond our assumption and hold for heavy-tailed Y 2 . Using the stability property, we prove a non-asymptotic lower bound for the coverage probability of prediction intervals constructed from the out-of-bag error of random forests. With another mild condition that is typically satisfied when Y is continuous, we also establish a complementary upper bound, which can be similarly established for the jackknife prediction interval constructed from an arbitrary stable algorithm. We also discuss the asymptotic coverage probability under assumptions weaker than those considered in previous literature. Our work implies that random forests, with its stability property, is an effective machine learning method that can provide not only satisfactory point prediction but also justified interval prediction at almost no extra computational cost.

1 Introduction

Random forests (RFs) is a successful machine learning method that serves as a standard approach to tabular data analysis and has good predictive performance [10, 7]. However, there is a big gap between the empirical effectiveness of RFs and the limited understanding of its properties. Most known theoretical results are established for variants of RFs not necessarily used in practice [5, 25, 22, 14, 30]. For the RF version implemented in packages like randomForest in R [21], little is known without strong assumptions [6, 26, 35]; RFs is notoriously difficult to analyze as a greedy algorithm. Here we show an important property for the RF used in practice (as well as for other variants) under realistic conditions.

1.1 Stability of random forests

The first main contribution of this work establishes the stability condition for the RF.

Theorem 1 (Stability of random forests, informal) . For independent and identically distributed (iid) training data points p X i , Y i q , i P t 1 , . . . , n u ' r n s and a test point p X,Y q , if the squared response Y 2 does not have a heavy tail, then the RF predictor RF B and any out-of-bag (OOB) predictor RF z i B predict similar values, i.e.,

where RF B results from the aggregation of all B base tree predictors, while RF z i B only those with the point p X i , Y i q excluded in training; ε n,B and ν n,B are small numbers depending on n and B .

This result is referred to as the stability of the RF because it indicates that no single training point is extremely important in determining RF B in a probabilistic sense. Theorem 1 relies on a recent important work that establishes the absolute stability (see below for a precise definition) of general bagged algorithms with bounded outputs [28]. We take advantage of the fact that the range of the RF output is conditionally dependent upon the maximal and minimal values of Y in the training set, and then we show in theory that the stability property of the RF is possible even if Y is marginally unbounded. To our knowledge, this is the first stability result established for the RF.

The technique used in our analysis requires that Y 2 not have a heavy tail (to make ε n,B and ν n,B small). Though arguably already mild, we conjecture that this condition might be further relaxed. As shown below, numerical evidence suggests that the light-tail assumption may not be necessary for RF stability, which could hold even when Y follows a heavy-tail distribution like the Cauchy distribution.

1.2 Random-forest prediction intervals

Stability is a crucial property of a learning algorithm. For example, stability has a deep connection with the generalization error of an algorithm [9, 19, 23]. Moreover, stability also turns out to be important in distribution-free predictive inference. In particular, an algorithm being stable justifies the jackknife prediction interval (PI), which otherwise has no coverage guarantee [3].

In this work, we show that stability makes it possible to construct a PI with guaranteed coverage from the OOB error of the RF. The OOB error is defined as R i ' | Y i ´ RF z i B p X i q| , i P r n s . A main reason why such a PI is appealing is that R i can be obtained almost without extra effort. For example, a one-shot training using the R package randomForest gives us an RF predictor RF B and all n OOB predictions RF z i B p X i q . So, from the computational point of view, a convenient way to construct a PI for a test point p X,Y q is of the form ' RF B p X q ˘ proper quantile of t R i u ' [17, 35].

The second main contribution of this work constructs such PIs and theoretically proves, under mild conditions, the non-asymptotic lower and upper bounds for the coverage probability.

Theorem 2 (Coverage lower bound, informal) . Under the same assumptions as in Theorem 1, and for α P p 0 , 1 q 1 , we have the following lower bound of coverage probability:

where r ¨ s is the ceiling function. Big O and other related notations are used in the usual way.

Theorem 3 (Coverage upper bound, informal) . If we further assume that Y is continuous, resulting in distinct prediction errors, then we also have the following upper bound:

As we detail below, the PIs we provide coverage guarantees for are neither the jackknife-with-stability interval discussed in [3], nor the jackknife+-after-bootstrap interval established in [18]. In our context, constructing the former needs n leave-one-out (LOO) predictors (rather than n OOB predictors), i.e., n additional RFs with each built on a training set of size n ´ 1 . Constructing the latter needs the explicit information of each RF z i B p¨q rather than the OOB prediction RF z i B p X i q for each X i only. Both these methods require additional, sometimes extensive, computation given current popular packages. In contrast, our results are operationally more convenient. After one-shot training, we obtain not only a point predictor RF B p¨q , but also a valid interval predictor at almost no extra cost. Under reasonable conditions, our results indicate that by slightly inflating (or deflating) the PI constructed from the r p n ` 1 qp 1 ´ α q s -th smallest R i , the coverage probability is guaranteed not to decrease (or increase) too much from the desired level of 1 ´ α . In fact, many numerical results, such as those in [17, 35], suggest that

Motivated by this fact, we further establish an asymptotic result of coverage for such PIs.

Theorem 4. (Asymptotic coverage, informal) In addition to the conditions in the above theorems, also suppose the prediction error | Y ´ RF B p X q| is continuous, and its cumulative distribution function (CDF) does not change too drastically for all sufficiently large n . Then

In [35], this asymptotic coverage was proved based on stronger assumptions. In particular, the true model is assumed to be additive such that ' Y ' f 0 p X q ` noise ' with the zero-mean noise independent of X , and RF B p X q is assumed to converge to f 0 p X q in probability. We do not require RF B to converge to anything in any sense when n Ñ8 . Technically, we need the family of prediction error CDFs be uniformly equicontinuous.

Based on our results, the RF seems to be the only one, among existing popular machine learning algorithms, that can provide both point and interval predictors with justification in such a convenient way. This makes the RF appealing, especially for tasks where the computational cost is a concern.

It is also worth noting that the upper-bound result is of interest in its own right. It can be generalized to jackknife PIs that are constructed from any stable algorithm; the result serves as a complement to the lower bounds established previously [3, 18].

Summarizing, we

theoretically prove that the (greedy) RF algorithm is stable when Y 2 does not have a heavy tail;

numerically show that RF stability may hold beyond the above light-tail assumption;

construct PIs based on the OOB error with finite-sample coverage guarantees: the lower bound of coverage does not need any additional assumption beyond stability; the upper bound needs an additional assumption, which is usually satisfied when Y is continuous;

provide the upper bound of coverage for jackknife PIs constructed from general stable algorithms, assuming distinct LOO errors; and

prove asymptotically exact coverage for RF-based PIs under weaker assumptions than those previously considered in published work.

2 Concepts of algorithmic stability

Stability stands at the core of this work. There are different types of stability, each of which is used to assess quantitatively how stable (in some certain sense) an algorithm is with respect to small variations in training data [9, 28, 4]. In a recent work [4], robust optimization is used to enhance the stability of algorithms in classification tasks. In [28], bagging is proved to be an efficient mechanism to stabilize algorithms in regression tasks. We focus on regression here. As will be made clear, the technique used in this work relies on the fact that the RF predictor in regression results from averaging tree predictors. However, the majority vote of tree predictors is used in classification, and new ideas are needed to analyze the RF stability in this setting. For our purposes, we introduce three levels of stability from strongest to weakest. The strongest version of stability, introduced in [28], does not depend on the data distribution, and may be referred to as 'absolute stability.'

Definition 1 (Absolute stability of algorithms) . For any dataset consisting of n ě 2 training points D ' tp X 1 , Y 1 q , . . . , p X n , Y n qu and any test point p X,Y q , an algorithm A is defined to be p ε, δ q -absolutely-stable if

for some ε, δ ě 0 , where ξ denotes the possible innate randomness in the algorithm (such as the node splitting procedure in the RF) and can be seen as a random variable uniformly distributed in r 0 , 1 s , ˆ f ' A p D ; ξ q is the predictor trained on D , and ˆ f ´ i ' A p D ´ i ; ξ q is the i th LOO predictor trained on D ´ i , i.e., D without the i th point p X i , Y i q . We might refer to the RF as both an algorithm (the learning procedure) and a predictor (the learned function) for simplicity.

Many bagged algorithms, in particular those with bounded predicted values, can achieve absolute stability with both ε and δ converging to 0, as long as n and the number of bags B go to infinity.

However, the predicted value of the RF is in general unbounded (for regression tasks considered in this work), and we are more interested in another type of stability, investigated in [9], and called out-of-sample stability [3]. For simplicity, we name it 'stability.' This notion of stability turns out to be important in validating a jackknife prediction interval.

Definition 2 (Stability of algorithms) . For iid training and test data, algorithm A is p ε, δ q -stable if

for some ε, δ ě 0 , where D,X, ˆ f, ˆ f ´ i are as defined above.

We will establish this type of stability for the derandomized RF defined below, where the datagenerating distribution is involved. To this end, we will use the methods in [28], which aim to provide absolute stability for bagged algorithms. Technically, we use such methods to first establish the 'conditional stability' of an algorithm with respect to given data.

Definition 3 (Conditional stability of algorithms) . Conditional on D and X , an algorithm A is defined to be p ε, δ q -conditionally-stable if

for some ε, δ ě 0 , where D,X, ˆ f, ˆ f ´ i are as defined above.

Once conditional stability is established for the derandomized RF algorithm, its stability can be consequently established by invoking

Stability of the derandomized RF provides the most essential ingredient for that of the practical RF, although the latter involves another type of stability, known as ensemble stability [18]. Ensemble stability justifies replacing the LOO predictor with the OOB predictor in (1). We may abuse the term 'stability' in the following when the OOB, rather than the LOO, predictor is used.

3 Stability of random forests

3.1 Basics of random forests

This work mainly considers using the RF to perform regression tasks, where the response Y P R can be unbounded. By construction, the RF predictor with B bags, denoted by RF B , is a bagged algorithm with the base algorithm being a tree, and RF B ' 1 B ř B b ' 1 TREE b , where TREE b is the b th tree predictor, trained on the b th bag r b , a bootstrapped sample of the training set D . The randomness in the tree predictor TREE originates from two independent sources: innate randomness ξ in the node splitting process and resampling randomness from the bag r . For the i th point, one can define the OOB RF predictor as RF z i B ' 1 B i ř B b ' 1 TREE b ˆ I t i R r b u , where I t¨u denotes the indicator function, and B i ' ř B b ' 1 I t i R r b u . Define p ' P p i P r q as the probability that the i th point is included in bag r . Then it is clear that B i ' Binomial p B, 1 ´ p q for all i . Wealso denote rf and rf z i as the derandomized versions of RF B and RF z i B , respectively. Precisely, rf ' E ξ,r r TREE s and rf z i ' E ξ,r r TREE | i R r s . It is worth noting that, by definition, RF z i B ‰ RF ´ i B for finite B , while rf z i ' rf ´ i as the derandomized RF results from the aggregation of an infinite number of trees. Since RF predictors are averages over tree predictors, the predicted values they output, given training set D , are bounded in r Y p 1 q , Y p n q s , where Y p 1 q and Y p n q are the minimum and maximum of t Y 1 , . . . , Y n u , respectively. We also let Z i ' | Y i | for all i , and denote the maximum as Z p n q . As a result, we have that

Remark 1. This is also true for RF z i B and RF B for any finite B . In fact, this is a distinctive feature of the RF, irrespective of the node splitting rule. Other regression methods do not necessarily have such a data-dependence bound. This observation helps to establish the conditional stability of the RF.

Remark 2. Practically, when n is large, one might think that the bound (2) is crude. On one hand, if we look for a bound valid for any finite n ě 2 , then there is not much room for improvement for small n . On the other hand, we do expect that the typical stability of the RF can go beyond the finite-sample guarantee provided by (2) when n is big, which is consistent with the numerical results shown below. A more informative bound for large n is worth future investigation.

There are several quantities that are useful in establishing the RF stability; they can be calculated explicitly and are listed below. First, it is well known that

Actually, p is monotonically decreasing for n ě 1 . Second,

as can be directly checked. Third, the moment generating function of B i is

In the following, we first perform the stability analysis for the derandomized RF (consisting of an infinite number of trees) and then extend the results to the practical finiteB case.

3.2 Derandomized random forests

The following theorem formalizes the conditional stability property for the derandomized RF, the proof of which is a direct result of Theorem 8 in [28], and is omitted here.

Theorem 5 (Conditional stability of derandomized random forests) . Conditional on training set D and test point p X,Y q , for the derandomized random forest predictor rf we have that

If δ p D,X q ě 1 , the statement is trivial, and we will focus on the case that δ p D,X q P p 0 , 1 q . We can now establish the stability property for the derandomized RF.

Theorem 6 (Stability of derandomized random forests) . For iid training and test data and ε ą 0 , the derandomized random forest predictor rf is stable with

This result follows directly from the conditional stability (6) by averaging over D and X . There is some freedom in choosing the dependence of ε on n . On one hand, in order to make sense of the word 'stability,' we do expect ε and ν to be small for large n . From (3) and (4), it is clear that the asymptotic behavior of ν is dominated by that of E r Z 2 p n q s{p ε 2 n q , which can be tuned by manipulating ε . For example, a matching convergence rate to 0 between ε and ν might be desirable, and one can then set ε ' O pp E r Z 2 p n q s{ n q 1 { 3 q if the scaling of E r Z 2 p n q s ' o p n q is known or can be inferred. On the other hand, we can fix ε to further investigate the relation between stability and the convergence-in-probability property of the RF. By (7), under the condition that E r Z 2 p n q s{ n Ñ 0 as n Ñ8 , one immediately comes to the conclusion that rf z i p X q ´ rf p X q converges to 0 in probability. Actually, a stronger conclusion can be drawn under the same condition.

Corollary 1. For iid training and test data, we have

Further assume that E r Y 2 s ă 8 . Then we have

Remark 3. The additional assumption that E r Y 2 s ă 8 is mild. Many commonly encountered random variables have a light tail and thus a finite second moment, irrespective of the detailed information of the distribution in question. Note that the bound (2) itself can be crude, and our result is expected to be valid even beyond this mild condition.

Remark 4. This result indicates that the difference is diminishing between rf and rf z i , built on n and n ´ 1 training data points, respectively. However, there is no indication that the derandomized rf p X q itself will converge to anything. This idea inspires the proposal of Theorem 11.

The proof of this result, as well as others below, will be deferred to the Appendix. So far, we have investigated the derandomized version of the RF, which is a limiting case and can be seen as consisting of an infinite number of trees, averaging out all kinds of possible randomness in the predictor construction process. In order to make the results more relevant to applied machine learning, the finiteB analysis for the RF is conducted below.

3.3 FiniteB random forests

We now consider the difference between RF B and RF z i B . We denote ξ ' p ξ 1 , . . . , ξ B q and r ' p r 1 , . . . , r B q as the corresponding sources of randomness in B trees. We also consider conditional stability first and then move to the stability of RF B .

Theorem 7 (Conditional stability of finiteB random forests) . Conditional on training set D and test point p X,Y q , for a random forest predictor RF B that consists of B trees, we have for ε ą 0 that

where δ is short for δ p D,X q as defined in (6) and g p p, δ, B q ' 2 p p `p 1 ´ p q δ 1 B q B .

Next, we consider the case of iid data and investigate the RF stability by averaging out the randomness in data. Note that Z p n q and δ are random and depend on the data distribution, while we are interested in a probability bound for | RF B p X q ´ RF z i B p X q| greater than a deterministic quantity, which is only a function of B and n . In this finiteB case, the stability of RF B cannot be directly obtained from its conditional stability as in the derandomized situation.

Theorem 8 (Stability of finiteB random forests) . Assume training points in set D and the test point p X,Y q are iid, drawn from a fixed distribution. For the random forest predictor RF B consisting of B trees and trained on D , we have

where ε n,B ' ř 3 i ' 1 ε i , and ν n,B ' ř 3 i ' 1 ν i . The pair of p ε 2 , ν 2 { λ q satisfies the derandomized RF stability condition (7) with λ ą 1 . Moreover, ε 1 ' ε 3 ' b 2 λ E r Z 2 p n q s ln p 1 ν 2 q{ B , ν 1 ' 2 ν 2 ` 2 P p Z 2 p n q ą λ E r Z 2 p n q sq , and ν 3 ' g p p, ν 2 , B q ` 2 P p Z 2 p n q ą λ E r Z 2 p n q sq .

On a high level, the establishment of this theorem relies on two observations: (i) the stability of the derandomized RF, so that the difference | rf p X q ´ rf z i p X q| is controlled, and (ii) the concentration of measure, so that the differences | RF B p X q ´ rf p X q| and | RF z i B p X q ´ rf z i p X q| are controlled. In order to make full sense of the word 'stability,' it is desirable that ε n,B and ν n,B can converge to 0. It is known that E r Y 2 s ă 8 suffices to ensure E r Z 2 p n q s ' o p n q [15, 13], and hence the stability of the derandomized RF. Now in the finiteB case, we need an additional distributional assumption to control the tail probability P p Z 2 p n q ą λ E r Z 2 p n q sq . It turns out that for typical light-tailed Y 2 , such a tail probability will converge to 0 as n Ñ8 . Technically, we can assume Y 2 to be sub-gamma [8]. Note that bounded and sub-Gaussian random variables are sub-gamma. Hence the sub-gamma assumption is not strong and can be satisfied by distributions underlying many real datasets.

Definition 4 (Sub-gamma random variables [8]) . A random variable W is said to be subgamma (on the right tail) with parameters p σ 2 , c q where c ě 0 , if ln E r e s p W ´ E r W sq s ď s 2 σ 2 2 p 1 ´ cs q for all s P p 0 , 1 { c q .

0.8

0.6

Predictionerror

0.5

OOBerror

0.6

0.2 0.3 0.4

Density

Density

0.4

0.2

0.1

8

0.0

-10

-5

0

5

-2

0

2

4

log10(AbsoluteDifference)

log10(Absolute Error)

Lemma 1. Suppose Y 2 is sub-gamma with parameters p σ 2 , c q with c ą 0 , and E r Z 2 p n q s ' a ln n with a ď c . For λ ą c { a , we have lim n Ñ8 P p Z 2 p n q ą λ E r Z 2 p n q sq ' 0 .

Remark 5. We have set c ą 0 above. If c ' 0 , then Y 2 is in fact sub-Gaussian, and the tail probability can be controlled similarly. If Y 2 is upper bounded by some constant M 2 , the stability analysis is even simpler, and there is no need to consider the tail probability at all, as we can use M 2 in place of Z 2 p n q in the conditional stability of the RF and then take expectation with respect to data.

Example 1. Consider Y 2 ' Exp p 1 q , the exponential distribution with scale parameter 1. It is known that Y 2 is sub-gamma with p σ 2 , c q ' p 1 , 1 q [8] , and E r Z 2 p n q s ' ř n i ' 1 1 i ' H n with H n P p γ ` ln n, γ ` ln p n ` 1 qq , where γ « 0 . 577 is Euler's constant. Hence H n ' ln n ` o p ln n q , and a straightforward calculation reveals that lim n Ñ8 P p Z 2 p n q ą λ E r Z 2 p n q sq ' 0 as long as λ ą 1 .

From such results, one can see that the vanishing tail probability is not a stringent condition. By taking this additional assumption, it is indeed possible that both ε n,B and ν n,B converge to 0.

Corollary 2. For the same setting as in Theorem 8, suppose Y 2 is sub-gamma with parameters p σ 2 , c q with c ą 0 and E r Z 2 p n q s ' a ln n with a ď c . Let λ ą c { a be a fixed number, and let B depend on n . Then for ε 2 that satisfies both ε 2 ' ω p a ln n { n q and ε 2 ' o p 1 q , and B ' Ω p ln 2 n q , we have lim n Ñ8 ε n,B ' lim n Ñ8 ν n,B ' 0 .

It is worth noting that there are multiple ways to let ε n,B and ν n,B approach 0, as the dependence of ε 2 , B , and even λ on n can all be manipulated. The point is that, theoretically, even the greedy RF can be stable with vanishing parameters. In practice, however, the stability of RF B seems to hold in broader situations where both the moment and tail assumptions on Y 2 can be relaxed.

3.4 Stability in practice and limitations of theory

We created a virtual dataset consisting of n ' 4000 points. We let Y be a standard Cauchy random variable, which is even without a well-defined mean. The feature vector X P R 3 is determined as X ' r 0 . 5 Y ` sin p Y q , Y 2 ´ 0 . 2 Y 3 , I t Y ą 0 u ` ζ s T where ζ is a standard normal random variable. We used 3000 of the points for training and 1000 of them as test points. Using the randomForest package with default setting (except letting B ' 1000 ), we had an output RF predictor RF B . We also aggregated corresponding tree predictors to have 3000 OOB predictors RF z i B . For each i P r 3000 s , we calculated the absolute difference | RF B p X q ´ RF z i B p X q| on 1000 test points to come up with a

density plot for such a difference, shown in Fig. 1. We also calculated 1000 absolute prediction errors | Y ´ RF B p X q| that are incurred by RF B on test points, and 3000 OOB errors | Y i ´ RF z i B p X i q| , each incurred by an OOB predictor RF z i B on its OOB point p X i , Y i q . The computation can be done within a few minutes on a laptop. The density plots of these two kinds of errors are also shown in Fig. 1. This example shows that the RF stability can be present beyond the realm guaranteed by the light-tail assumption. As mentioned above, this is because the bound (2) can be conservative when n is large. We hope our results can inspire future study towards a more informative bound. Also, the similarity between the prediction error and the OOB error in this heavy-tail case indicates that the RF-based PIs analyzed below can find more applications in practice than justified by the current theory.

4 Random-forest prediction intervals

4.1 Comparison with related methods

With the stability property of the RF, it is possible to construct PIs with finite-sample guaranteed coverage. Recent years have witnessed the development of distribution-free predictive inference [1] with the full [33, 27], split [24, 31, 20], and jackknife+ [3, 32] conformal prediction methods being three milestones. The full conformal method is computationally prohibitive when used in practice. The split method greatly reduces the computational cost but fails to thoroughly extract the available information of training data. The jackknife+ ( J+ ) method maximizes the usage of data at a computational cost in between those of full and split methods. In [18], jackknife+-after-bootstrap ( J+aB ) was proposed for bagged algorithms to achieve the same goal as in J+, while the training cost can be further reduced. However, the number of bags B is required to be a Binomial random variable, which might seem unnatural. It turns out that by further imposing the assumption of ensemble stability (which is essentially the concentration of resampling measure), J+aB can still have guaranteed coverage with a fixed B . Ensemble stability is defined for bagged algorithms. It measures how typical a bootstrap sample is, and is different from the algorithmic stability that quantifies the influence of removing one training point. If algorithmic stability is also imposed, then not only J+aB, but also jackknife can provide guaranteed coverage, which is otherwise impossible [29, 3].

Conceptually, the J+ approach and its variants under stability conditions are particularly relevant to this work. As the stability we establish for the RF contains both ensemble and algorithmic components, we will generally refer to the J+aB method with both ensemble and algorithmic stability as J+aBS and the jackknife method with algorithmic stability as JS . Our method might be best described as 'jackknife-after-bootstrap-with-stability ( JaBS )' tailored for the RF, which is different from both JS and J+aBS. Our method requires the least effort of computing as only one output predictor is needed, while all others require at least n output predictors.

There also exist RF-based PIs [17, 35] that are essentially of the jackknife-after-bootstrap ( JaB ) type and almost identical to ours practically when ε is small and n equals the size of a typical dataset. However, without stability, there is, in general, no guarantee for the coverage of such PIs, although the asymptotic coverage 1 ´ α can be established based on strong assumptions [35]. We take advantage of the stability of the RF algorithm to establish the lower bound of coverage in Theorem 9 below. An upper bound is established in Theorem 10 with an additional mild assumption. We also propose a weaker assumption for asymptotic coverage in Theorem 11.

We compare these relevant methods to ours in Table 1 and Table 2, where the RF is set as the working algorithm for all methods and p ε, ν q is a general pair of stability parameters. We define q n,α t R i u , q ` n,α t R i u , q ´ n,α t R i u , and q 1 n,α t R i u as follows. Given t a 1 , . . . , a n u ,

where t ¨ u is the floor function. Let R LOO i ' | Y i ´ RF ´ i B p X i q| be the LOO error, where RF ´ i B is trained without the i th training point, and by definition RF ´ i B ‰ RF z i B .

In Table 1, we list the corresponding PI constructed from each method and the output predictors of each method. The number of output predictors directly reflects the computational cost. It is

worth noting that acquiring the LOO predictor RF ´ i B needs substantial computation. In packages like randomForest , aggregating tree predictors to obtain the OOB predictor RF z i B also needs extra computation. However, the predicted value RF z i B p X i q can be obtained immediately by calling the predict pq function. The fact that the value of RF z i B p X q on a test point is not needed further reduces the computational cost of JaB and our method, which only need one output RF predictor, and are more favorable computationally.

In Table 2, we list the coverage of the PI constructed from each method, as well as the additional conditions (beyond iid data) needed to achieve the coverage. Note that J+ does not require any additional conditions to achieve the coverage lower bound 1 ´ 2 α , but J+aB requires that the number of trees B be a Binomial random variable. For JS, J+aBS, and our method, stability is needed to achieve the coverage lower bound 1 ´ α ´ O p ? ν q . With additional mild assumptions, the coverage upper bound and asymptotic coverage of our method can be established. However, there is no guarantee of coverage for JaB without strong assumptions.

In summary, our theoretical work provides a series of coverage guarantees to a computationally feasible method for constructing PIs based on the RF algorithm. In the following, we will establish the lower and upper bound of coverage, as well as the asymptotic coverage.

4.2 Non-asymptotic coverage guarantees

Theorem 9 (Coverage lower bound) . Suppose the RF predictor RF B satisfies the stability condition as in Theorem 8. Then we have for a test point p X,Y q that

This result is established by starting from the analysis of an imaginary extended dataset D ' D Ytp X,Y qu , where the test point is assumed to be known. We denote p X,Y q as p X n ` 1 , Y n ` 1 q for convenience. For all points in D , consider the derandomized RF predictor r rf z i that is built on n data

points without the i th point in D , i P r n ` 1 s . One can then define the OOB error r r i ' | Y i ´ r rf z i | . Since all data are iid, we have that P p r r n ` 1 ď q n,α t r r i uq ě 1 ´ α , where q n,α t r r i u is the r p 1 ´ α qp n ` 1 q s -th smallest value of t r r 1 , . . . , r r n u . Next, notice r r n ` 1 ' | Y n ` 1 ´ r rf zp n ` 1 q p X n ` 1 q| ' | Y n ` 1 ´ rf p X n ` 1 q| by the definitions of r rf zp n ` 1 q and rf . By concentration of measure, rf p X n ` 1 q can be approximated by RF B p X n ` 1 q , and thus r r n ` 1 can be roughly replaced with | Y n ` 1 ´ RF B p X n ` 1 q| , which is desired. Then by stability of rf , t r r i u can be approximated by t r i ' | Y i ´ rf z i p X i q|u . Although t r i u is still unavailable in practice, by applying the idea of concentration of measure again, t r i u can be further approximated by t R i u , which is accessible given D . Eventually, we can bound | Y n ` 1 ´ RF B p X n ` 1 q| in terms of t R i u . The approximations are accounted for by the stability parameters in Theorem 8.

If we further assume that there are no ties among t r r i u , i P r n ` 1 s , a typical case when Y is continuous, then we can also establish the upper bound of coverage.

Theorem 10 (Coverage upper bound) . Suppose there are no ties in t r r i u , i P r n ` 1 s , and the RF predictor RF B satisfies the stability condition as in Theorem 8. Then

The upper bound can be established because if there are no ties among r r 1 , . . . , r r n ` 1 , then P p r r n ` 1 ď q n,α t r r i uq ď 1 ´ α ` 1 n ` 1 . The apparent symmetry between the lower and upper bound originates from the fact that they both are established by using the RF stability once and the concentration of measure twice. Note that this idea can be applied to JS intervals for an arbitrary stable algorithm in exactly the same way, providing a complement to the lower bound for JS intervals established in [3].

Corollary 3 (Coverage upper bound for jackknife-with-stability intervals) . Let ˆ f be a predictor trained on n iid data points and ˆ f ´ i be the LOO predictor without the i th point. Suppose ˆ f is stable with P p| ˆ f p X q ´ ˆ f ´ i p X q| ą ε q ď ν , and the LOO errors are distinct on the extended training set that includes an iid test point p X,Y q . Then we have P ´ | Y ´ ˆ f p X q| ď q n,α t r i ´ ε u ¯ ď 1 ´ α ` 1 ` 2 ? ν , where r i are the LOO errors on the original training set.

n ` 1

4.3 Asymptotic coverage guarantee

As shown above, the stability parameters p ε n,B , ν n,B q can vanish when n Ñ 8 . It is reasonable to expect that P p Y P RF B p X q ˘ q n,α t R i uq Ñ 1 ´ α in this limit, as is consistent with numerous empirical observations [17, 35]. However, to achieve this goal, it seems that more assumptions are unavoidable. In [35], the guaranteed coverage of the JaB method is established by assuming that RF B p X q converges to some f 0 p X q in probability as n Ñ8 , where f 0 is the true regression function of an additive model that generates the data. We show that this can be done under weaker conditions.

Theorem 11 (Asymptotic coverage) . Denote F n as the CDF of | Y ´ RF B p X q| . Suppose t F n u n ě n 0 is uniformly equicontinuous for some n 0 . Then P p Y P RF B p X q ˘ q n,α t R i uq Ñ 1 ´ α as n Ñ8 when conditions in Theorem 9, Theorem 10, and Corollary 2 are satisfied.

Remark 6. Intuitively, using errors from RF z i B that are trained on n ´ 1 points to approximate those from RF B , trained on n points, we only need this approximation to be exact asymptotically. There is no need for RF B itself to converge to anything. This is one major conceptual difference between our work and [35], and it is in this sense that our assumption is weaker. Practically, this kind of PI is recommended as it does not involve p ε n,B , ν n,B q , and has great performance on numerous datasets.

5 Conclusion

In this work, for the first time, we theoretically establish the stability property of the greedy version of random forests, which is implemented in popular packages. The theoretical guarantee is based on a light-tail assumption of the marginal distribution of the squared response Y 2 . However, numerical evidence suggests that this stability could persist in much broader situations. Based on the stability property and some mild conditions, we also establish finite-sample lower and upper bounds of coverage, as well as the exact coverage asymptotically, for prediction intervals constructed from the out-of-bag error of random forests, justifying random forests as an appealing method to provide both point and interval prediction simultaneously.

Acknowledgments and Disclosure of Funding

Much of the work was completed while Yan Wang was a PhD student in the Department of Statistics at Iowa State University. This research was supported in part by the US National Science Foundation under grant HDR:TRIPODS 19-34884.

References

Anastasios N Angelopoulos, Stephen Bates, et al. Conformal prediction: A gentle introduction. Foundations and Trends® in Machine Learning , 16(4):494-591, 2023.

Arthur Asuncion and David Newman. UCI Machine Learning Repository, 2007.

Rina Foygel Barber, Emmanuel J Candes, Aaditya Ramdas, and Ryan J Tibshirani. Predictive inference with the jackknife+. The Annals of Statistics , 49(1):486-507, 2021.

Dimitris Bertsimas, Jack Dunn, and Ivan Paskov. Stable classification. The Journal of Machine Learning Research , 23(1):13401-13453, 2022.

Gérard Biau. Analysis of a random forests model. The Journal of Machine Learning Research , 13(1):1063-1095, 2012.

Gérard Biau, Luc Devroye, and Gäbor Lugosi. Consistency of random forests and other averaging classifiers. The Journal of Machine Learning Research , 9(9):2015-2033, 2008.

Gérard Biau and Erwan Scornet. A random forest guided tour. Test , 25:197-227, 2016.

Stéphane Boucheron, Gábor Lugosi, and Pascal Massart. Concentration Inequalities: A Nonasymptotic Theory of Independence . Oxford University Press, 2013.

Olivier Bousquet and André Elisseeff. Stability and generalization. The Journal of Machine Learning Research , 2:499-526, 2002.

Leo Breiman. Random forests. Machine Learning , 45(1):5-32, 2001.

Thomas F Brooks, D Stuart Pope, and Michael A Marcolini. Airfoil Self-Noise and Prediction . National Aeronautics and Space Administration, Office of Management, 1989.

Andrea Coraddu, Luca Oneto, Aessandro Ghio, Stefano Savio, Davide Anguita, and Massimo Figari. Machine learning approaches for improving condition-based maintenance of naval propulsion plants. Proceedings of the Institution of Mechanical Engineers, Part M: Journal of Engineering for the Maritime Environment , 230(1):136-153, 2016.

José R Correa and Matías Romero. On the asymptotic behavior of the expectation of the maximum of iid random variables. Operations Research Letters , 49(5):785-786, 2021.

Misha Denil, David Matheson, and Nando De Freitas. Narrowing the gap: Random forests in theory and in practice. In International Conference on Machine Learning , pages 665-673. PMLR, 2014.

Peter J Downey. Distribution-free bounds on the expectation of the maximum with scheduling applications. Operations Research Letters , 9(3):189-201, 1990.

Francesca Grisoni, Viviana Consonni, Marco Vighi, Sara Villa, and Roberto Todeschini. Investigating the mechanisms of bioconcentration through QSAR classification trees. Environment International , 88:198-205, 2016.

Ulf Johansson, Henrik Boström, Tuve Löfström, and Henrik Linusson. Regression conformal prediction with random forests. Machine Learning , 97:155-176, 2014.

Byol Kim, Chen Xu, and Rina Barber. Predictive inference is free with the jackknife+-afterbootstrap. Advances in Neural Information Processing Systems , 33:4138-4149, 2020.

Aryeh Kontorovich. Concentration in unbounded metric spaces and algorithmic stability. In International Conference on Machine Learning , pages 28-36. PMLR, 2014.

Jing Lei, Max G'Sell, Alessandro Rinaldo, Ryan J Tibshirani, and Larry Wasserman. Distribution-free predictive inference for regression. Journal of the American Statistical Association , 113(523):1094-1111, 2018.

Andy Liaw and Matthew Wiener. Classification and regression by randomForest. R News , 2(3):18-22, 2002.

Yi Lin and Yongho Jeon. Random forests and adaptive nearest neighbors. Journal of the American Statistical Association , 101(474):578-590, 2006.

Andreas Maurer and Massimiliano Pontil. Concentration inequalities under sub-Gaussian and sub-exponential conditions. Advances in Neural Information Processing Systems , 34:7588-7597, 2021.

Harris Papadopoulos, Kostas Proedrou, Volodya Vovk, and Alex Gammerman. Inductive confidence machines for regression. In European Conference on Machine Learning , pages 345-356. Springer, 2002.

Erwan Scornet. Random forests and kernel methods. IEEE Transactions on Information Theory , 62(3):1485-1500, 2016.

Erwan Scornet, Gérard Biau, and Jean-Philippe Vert. Consistency of random forests. The Annals of Statistics , 43(4):1716-1741, 2015.

Glenn Shafer and Vladimir Vovk. A tutorial on conformal prediction. Journal of Machine Learning Research , 9(3), 2008.

Jake A Soloff, Rina Foygel Barber, and Rebecca Willett. Bagging provides assumption-free stability. arXiv preprint arXiv:2301.12600 , 2023.

Lukas Steinberger and Hannes Leeb. Conditional predictive inference for stable algorithms. The Annals of Statistics , 51(1):290-311, 2023.

Cheng Tang, Damien Garreau, and Ulrike von Luxburg. When do random forests fail? Advances in Neural Information Processing Systems , 31, 2018.

Vladimir Vovk. Conditional validity of inductive conformal predictors. In Asian Conference on Machine Learning , pages 475-490. PMLR, 2012.

Vladimir Vovk. Cross-conformal predictors. Annals of Mathematics and Artificial Intelligence , 74(1):9-28, 2015.

Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic Learning in a Random World . Springer Science & Business Media, 2005.

I-C Yeh. Modeling of strength of high-performance concrete using artificial neural networks. Cement and Concrete Research , 28(12):1797-1808, 1998.

Haozhe Zhang, Joshua Zimmerman, Dan Nettleton, and Daniel J Nordman. Random forest prediction intervals. The American Statistician , 74(4):392-406, 2020.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies the random forests stablity for regression problem, and the authors presents theoretical analysis on the upper and lower boudns for the coverage probability of prediction intervals constructed from the out-of-bag error of random forests. The theoretical guarantee is based on a light-tail assumption of the marginal distribution of the squared response. \n\n------------------after response---------------------------------\n\nAfter reading the authors' response, I do not think the authors answer my concerns, in particularly for the novelties and significances. \n\nAs a theoreical work, it is very important to evaluate from theoretical novelties and techniques, while I find some inremental results based on well-known techniques.\n\nI do not find any experiments, and I do not know why the authors could claim that \"Our work applies to many variants of random forests ... which makes it particularly relevant in theory and practice\".\n\nSTRENGTHS:\n1) It is an interesting problem on the theoretical understanding of random forests.\n\n2) Some theoretical results on the convergence probability of prediction intervals constructed from the out-of-bag error of random forests. \n\n3) Limited theoretical techinical contributions\n\nWEAKNESSES:\n1) The problem is not very clear. The authors should first present the studied problem, i.e. the original ranfom forests for regression, or randomf forest interval, or  prediction intervals constructed from the out-of-bag error of random forests. It is very confused to understand the main contributions in the current submission and relevant work. For completeness, it would be better to present the detailed algorithm, rather than finding some other research work for other readers.\n\n2) The main conclusions are not clear. The main contribution of this work is the stability of random forests in Theorem 1. Generally, a theoretical work concerns seriously the convergence rate of stability, and how tight of this rate. It would be better to present the specific expression for \\nu_{n,B}, and make necessary discussions. What's factors affect the stability rate?\n\n3) Some important definitions and notions are missing. For example, where is the definition of \"light tail\"， whichi is the basic assumption in main theoretical results. How to characaterize light tail and its relevant factors.\n\n4) The authros should clearify the novelty and significance of the main results, for example, how about the theoretical new insghts on the technical proof in this work. As an pure theoretical problem, it is importnat to present some new technical proof, rather than simple extension of the current techiniques. What's the siginificance of the main results, is it possible to present some pratical guidance and suggest some new algorithms.\n\n5) The authors should have a good background on the theoretical analysis on random forests, for example, \nG. Biau, L. Devroye, G. Lugosi, Consistency of random forests and other averaging classifiers, JMLR 2008.\nM. Denil, D. Matheson, N. De Freitas, Narrowing the gap: random forests in theory and in practice, ICLM2014.\nW. Gao, F. Xu and Z.-H. Zhou. Towards convergence rate analysis of random forests for classification. AIJ, 2022.\n\nQUESTIONS:\n1) The problem is not very clear. The authors should first present the studied problem, i.e. the original ranfom forests for regression, or randomf forest interval, or  prediction intervals constructed from the out-of-bag error of random forests. It is very confused to understand the main contributions in the current submission and relevant work. For completeness, it would be better to present the detailed algorithm, rather than finding some other research work for other readers.\n\n2) The main conclusions are not clear. The main contribution of this work is the stability of random forests in Theorem 1. Generally, a theoretical work concerns seriously the convergence rate of stability, and how tight of this rate. It would be better to present the specific expression for \\nu_{n,B}, and make necessary discussions. What's factors affect the stability rate?\n\n3) Some important definitions and notions are missing. For example, where is the definition of \"light tail\"， whichi is the basic assumption in main theoretical results. How to characaterize light tail and its relevant factors.\n\n4) The authros should clearify the novelty and significance of the main results, for example, how about the theoretical new insghts on the technical proof in this work. As an pure theoretical problem, it is importnat to present some new technical proof, rather than simple extension of the current techiniques. What's the siginificance of the main results, is it possible to present some pratical guidance and suggest some new algorithms.\n\nLIMITATIONS:\nThis is a pure theoretical work", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nRandom forests are one of the most used Machine Learning methods. Its standard variant (for regression) takes the following form. GIven a random sample $D=(X_i,Y_i)_{i\\leq n}$ of covariate/response pairs, one takes $B$ bootsrapped samples from $D$ and trains a tree regressor on each boostrapped sample (using one of many possible criteria). The estimated regression function $RF(x)$ is the average of the $B$ tree regressors on $x$. \n\nTheorem 1 of the present paper is a kind of stability property for $RF$ vis a vis its \"out-of-bag\" variant $RF^{\\backslash i}$. Here $i\\in [n]$ and $RF^{\\backslash i}$ is the version of $RF$ where bootstrapped samples not containing the $i$-th sample point are discarded. The paper then shows that one can use this stability property to build predictive intervals from the \"OOB residuals\" $|Y_i-RF^{\\backslash i}(X_i)|$. Theorem 2 proves that these intervals, after a slight enlargement, provide good coverage at near-nominal-levels, whereas Theorem 3  gives exact $1-\\alpha$ coverage in the limit, under suitable assumptions. \n\nOne of the main assumptions the authors impose for these last results is that the response variables be light-tailed. However, a small set of experiments suggests that similar properties would hold even for very heavy tailed covariates.\n\nThe present paper joins string of recent works dealing with uncertainty quantification for ML methods without resorting to a calibration sample (in which case one could use conformal prediction). The authors argue that main distinguishing feature of Theorems 2 and 3 is that their predictive intervals are much less demanding computationally. As computing the OOB predictors comes as a byproduct of the RF computations. By contrast, related work (eg. on jacknife type estimators) require the regression method to be rerun several times on leave-one-out samples. \n\nLet me also say a few words about proofs. The idea is to first explore the $B\\to +\\infty$ limit of $RF$ and $RF^{\\backslash i}$ and prove stability in that setting, via arguments from arXiv:2301.12600.\n\nSTRENGTHS:\nAs noted above, the paper obtains a lightweight method to compute (nearly-)valid prediction intervals from random forests, which are often used in practice. This means that the result is significant (though see the next field). It also seems to be original. The exposition is fairly clear.\n\nWEAKNESSES:\n* Mathematically, it seems that much of the work behind the paper comes from arXiv:2301.12600 by Soloff, Barber and Willett. \n* The nonasymptotic Theorem 2 is a bit unsatisfying in that both the interval length and the coverage are subject to errors that are hard to quantify in practice (still, since people will use RF anyway, it's nice to have some result of this kind). \n* The results do not give a good bound on how large $B$ needs to be in order for the method to work well. This is clearly related not just to the maximum $Z_{(n)}$, but to how small an error one wants to allow when defining the interval.  \n* One very minor comment: Theorem 3.4 from arXiv:2301.12600 (quoted in line 163) should be Theorem 9 (the numbering seems to have changed from v1).\n\nQUESTIONS:\n1) It seems from reading arXiv:2301.12600 that $Z_{(n)}$ could be replaced with the range $\\max_{1\\leq i<j\\leq n}|Y_i-Y_j|$ throughout the analysis. Is this true? One advantage of the range is that it is centered. \n\n2) The authors obtain what is called a \"marginal coverage\" guarantee: if one single extra point $(X,Y)$ is given, then everything work. A stronger guarantee would be this: letting $\\mu$ denote the distribution of $(X,Y)$\n\n$$P\\{\\mu\\{(x,y): |y-RF(x)|\\leq  q_\\alpha(R_i) + \\epsilon\\}\\geq 1-\\alpha-\\nu.$$\n\nCould this stronger guarantee be obtained via the present methods. \n\n3) As noted in a previous field, computing how large $B$ needs to be seems to be a pressing problem. Checking the bound in Theorem 12 in arXiv:2301.12600 (v2), it seems that the present paper skips the variance-based proof for the variability of $RF_B$ and goes straight to the range-based bound. This means that the parameter $\\epsilon_{n,B}$ in Theorem 2 is a sum of two parts: one that would be there even if $B=+\\infty$, and a second term that comes from finite $B$. It seems to me, however, that this second term could be controlled via suitable empirical concentration inequalities such as Lemma E.4 in https://arxiv.org/pdf/2212.09900.pdf: in practice, this would mean that the second error term just mentioned could be controlled for finite B. This, in turn would make the method a bit heavier, but more quantifiable. Could the authors comment on this?\n\n4) Regarding Theorems 3 and 10: it would seem that a sufficient condition for the theorem to hold is that \n$Y-\\mathbb{E}[Y\\mid X]=:\\eta$ satisfies:\n$$\\lim_{h\\searrow 0}\\mathbb{E}[\\sup_{x,y\\in\\mathbb{R}\\,:\\,|x-y|\\leq h}|F_{\\eta\\mid X}(x)-F_{\\eta\\mid X}(y)|]=0.$$\nwhere $F_{\\eta\\mid X}$ is the conditional cdf of $\\eta$ given $X$. This kind of condition is very natural: eg. it holds if the density of $\\eta\\mid X$ is uniformly bounded. Would it indeed be sufficient to require this condition on $\\eta$?\n\nLIMITATIONS:\nThere is no explicit discussion of limitations, nor do I think one would be necessary.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

comments: SUMMARY:
In this paper the authors considers the issue of stability of the often used in practice Random Forest algorithm and provide theoretical bounds on the $\varepsilon$-stability upto an order of $O_{\mathbb{P}}(|Y|^2_{(n)}/n)$ (i.e. the largest in magnitude observation) when fitting the method with $n$-iid sample points $(X_i,Y_i)$. A light tailed assumption on $Y^2$ thereby yields suitable control over the asymptotic behavior of $|Y|^2_{(n)}/n$. Further, the stability results are used to derive $n$-dependent lower and upper bounds (under increasingly more assumptions) for the coverage probability of prediction intervals constructed from out-of-bag error of random forests. In comparison to many other results in the literature, this paper works with a practical version of random forests.

STRENGTHS:
1. In comparison to many other results in the literature, this paper works with a more practical version of random forests.

WEAKNESSES:
1. It was not clear whether the stability bounds were optimal or could be improved.
2. The presentation usually benefits from working under a single set of assumptions instead of increasingly more assumptions (however this is a minor point).

QUESTIONS:
If one assumes that $Y=f(X)+\eta$ for $\eta \sim F$ for some suitable light tailed distribution and $f\in \mathcal{F}$ in some classical function class, can one have modified rates for stability bounds depending on $F$ and $\mathcal{F}$?

LIMITATIONS:
None noted
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents new and strong set of results on stability of (greedy version of) random forests. Theoretical (resp. numerical) evidence is provided to support stability for light-tail (and heavy-tail) assumptions on marginal distribution of squared response. New finite sample upper and lower bounds are provided for prediction intervals constructed from OOB effort of random forests. The paper can be regarded as a demonstrative work that justifies the merit of random forests for both point and interval prediction.\n\nSTRENGTHS:\nThe style of results and overall content of paper are very appealing. I like the way in which stability and prediction interval results were informally stated first and then rigorously discussed later. A clear review of algorithmic stability concepts helped me understand the proof ideas.\n\nWEAKNESSES:\n-- I think the writing can be slightly improved by articulating the use of absolute stability results for bagged algorithms. \n-- The transition from derandomized version of RF to finite-B case through the route of conditional stability analysis is a bit abrupt and needs more explanation, particularly with regard to concentration of measure. \n-- The limitation/extension of theory to heavy-tailed case is interesting but some comments about the key bottlenecks in the proof technique that would need to be overcome to achieve such a generalization would make the contribution stronger. Also where does the theory break? The experimental results are not very clear in this regard. \n-- On the prediction interval part, Table 1 can be enhanced and repositioned to include computational advantages as well. Finally, the discussion around “jackknife-after bootstrap-with-stability (JaBS)” is a bit hard to follow since the discussion on J+aB and J+aBS is intertwined with the intuition on how subsequent results in the paper build on stability results – it is desirable to streamline this discussion a bit. \n-- please consider citing and discussing relevant work in OR on optimal classification/prescriptive trees by Bertsimas and co-authors as well as their work on Stable Classification. Your approach and focus is different, but given the stated goal of putting the stability of random forests on a stronger footing, it may be worthwhile to make a connection.\n\nQUESTIONS:\nPlease see my suggestions above.\n\nLIMITATIONS:\nEverything on this front seems reasonable.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}