SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \Delta (A, 1, b) = \int_ {\ell} ^ {r} \left(t - \frac {\ell + r}{2}\right) ^ {2} \mathrm {d} t - \int_ {\ell} ^ {b} \left(t - \frac {\ell + b}{2}\right) ^ {2} \mathrm {d} t - \int_ {b} ^ {r} \left(t - \frac {b + r}{2}\right) ^ {2} \mathrm {d} t \tag {13} \\ = \frac {1}{1 2} (r - \ell) ^ {3} - \frac {1}{1 2} (b - \ell) ^ {3} - \frac {1}{1 2} (r - b) ^ {3} = \frac {3}{4 8} (r - \ell) ^ {3} \\ \end{array}
$$
Combining (12) and (13) we know that Assumption 2.2 is satisfied with  $\lambda = 3 / 4$ .
As shown by Example 2.1, when  $f^{*}$  is a univariate linear function and  $\{x_{i}\}_{i = 1}^{n}$  are from a uniform distribution, the error bound in (11) reduces to  $\widetilde{O}(n^{-2/3})$ . This rate matches the lower bound for any partition-based estimator (see e.g. [13] Chapter 4), hence showing that the upper bound in (11) cannot be improved (by more than a log factor) via any refined analysis. To verify the rate  $\widetilde{O}(n^{-2/3})$ , we conduct a simulation shown by Figure 1. In particular, Figure 1 presents the log-log plot of the prediction error of CART versus different sample sizes, where the depth  $d$  is taken as stated above (11). A reference line of slope  $-2/3$  is also shown. It can be seen that the slope of the line for CART is also roughly  $-2/3$ , which is consistent with the rate  $\widetilde{O}(n^{-2/3})$  derived in Example 2.1.
Figure 1: Prediction error of CART versus sample size  $n$  for a univariate linear model.
There is a subtle difference between the lower bound in [13] Chapter 4 and the upper bound in Theorem 2.3. The lower bound in [13] requires that the splitting points of the partition are independent of the training data, while for CART this condition is not satisfied.
3 Function classes satisfying SID condition
In this section, we introduce a class of sufficient conditions that facilitate the easy verification of the SID condition for a broad range of functions. Specifically, we focus on the case when  $f^{*}$  has an additive structure:
$$
f ^ {*} (u) = f _ {1} ^ {*} \left(u _ {1}\right) + f _ {2} ^ {*} \left(u _ {2}\right) + \dots + f _ {p} ^ {*} \left(u _ {p}\right) \tag {14}
$$
where  $f_{j}^{*}$ 's are univariate functions. Note that additive model has also been assumed in prior works such as [19, 17, 16] for the study of CART. If we only assume that each component function  $f_{j}^{*}$  has bounded total variation, then only a slow rate  $O(\log^{-1}(n))$  is known [17]. In the following, we discuss a few sufficient conditions concerning the component functions  $f_{j}^{*}$  under which  $f^{*}$  satisfies the SID condition, consequently guaranteeing a faster convergence rate as indicated by Theorem 2.3.
First, we introduce a key class of univariate functions that satisfy a special integral inequality.
Definition 3.1 (Locally Reverse Poincaré Class) For a univariate differentiable function  $g$  on an interval  $Q \subset \mathbb{R}$ , we say it belongs to the Locally Reverse Poincaré (LRP) class on  $Q$  with parameter  $\tau$  if for any subinterval  $[a, b] \subseteq Q$ ,
$$
\left(\int_ {a} ^ {b} \left| g ^ {\prime} (t) \right| \mathrm {d} t\right) ^ {2} \leq \frac {\tau^ {2}}{b - a} \inf  _ {w \in \mathbb {R}} \int_ {a} ^ {b} | g (t) - w | ^ {2} \mathrm {d} t \tag {15}
$$
We use the notation  $LRP(Q, \tau)$  to denote the class of all such functions.
It is worth noting that for any given univariate differentiable function  $g$  and a fixed interval  $[a, b]$ , there exists a constant  $\tau$  such that the reverse of inequality (15) holds true, as established by the Poincaré inequality. The condition (15) requires a uniform constant  $\tau > 0$  such that the reverse of the Poincaré inequality holds true. Note that the LHS of (15) is equivalent to the square of the total variation of  $g$  on  $[a, b]$ , and the RHS is a constant multiple of the conditional variance of  $g$  on  $[a, b]$  (under the uniform distribution). Intuitively, this condition can be satisfied when the function  $g$  exhibits a clear "trend" within any interval  $[a, b]$  rather than being fuzzy fluctuations. We give a few examples below.
Example 3.1 (Strongly increasing function) Let  $g$  be a strictly increasing function on  $[0,1]$  with  $c_2 \geq g'(t) \geq c_1 > 0$  for all  $t \in [0,1]$ . Then  $g \in LRP([0,1], 2\sqrt{3}c_2/c_1)$ .
Example 3.1 provides a direct generalization of the simple linear function showcased in Example 2.1. Note that this example was also discussed in [10]. In Example 3.1, since  $g'(t)$  is consistently bounded away from zero, verifying the inequality 15 becomes straightforward. Below we present a more nuanced example where  $g'(t)$  may equal zero at certain point  $t \in [0,1]$ .
Example 3.2 (Smooth and strongly convex function) Let  $g$  be a  $L$ -smooth and  $\sigma$ -strongly-convex function on  $[0,1]$  for some  $L > \sigma > 0$ , i.e.,
$$
\frac {\sigma}{2} (t - s) ^ {2} \leq g (t) - g (s) - g ^ {\prime} (s) (t - s) \leq \frac {L}{2} (t - s) ^ {2} \tag {16}
$$
for all  $t, s \in [0,1]$ . Then  $g \in LRP([0,1], 110(L / \sigma))$ .
Note that for a smooth and strongly convex function  $g$ , the gradient  $g'(t)$  can be zero at some  $t \in [0,1]$ . But the strong convexity condition guarantees that there is at most one point  $t$  where  $g'(t) = 0$ , and the inequality (15) can still be satisfied. Additional details can be found in Appendix B.
Example 3.3 (Polynomial with degree  $r$ ) There exists a constant  $C_r$  that only depends on  $r$  such that for any univariate polynomial  $g$  of degree at most  $r$ ,  $g \in LRP(\mathbb{R}, C_r)$ .
Although the set of polynomials (with a degree at most  $r$ ) appears to be complicated, it is a finite-dimensional linear space. Therefore the differentiation is a bounded linear operator between finite dimensional normed linear spaces. As a result, the conclusion of Example 3.3 can be proved via a variable transformation argument. See Appendix B for details.
Examples 3.1 - 3.3 demonstrate that the LRP condition can be readily verified for various well-known function classes, but the relationship between the LRP class and the SID condition remains unclear. Below we present the first main result in this section, which establishes a connection between the LRP class and the SID condition under the additive model.
Proposition 3.1 Suppose Assumption 2.1 holds true, and  $f^{*}$  has an additive structure as in (14). If  $f_{k}^{*} \in LRP([0,1],\tau)$  for all  $k \in [p]$ , then Assumption 2.2 is satisfied with  $\lambda = 4\theta /(p\tau^2\bar{\theta})$ .
As shown by Proposition 3.1, for the additive model (14), when the basic conditions in Assumption 2.1 are satisfied, the SID condition holds true for  $f^{*}$  as long as each component function is in an LRP class. Particularly, the SID condition is satisfied for component functions from Examples 3.1 - 3.3. Note that the coefficient  $\lambda$  in Proposition 3.1 is proportional to  $1 / p$ , which implies that it can be very small when  $p$  is large. In particular, with this  $\lambda = \Theta(1 / p)$  (with  $\underline{\theta}, \bar{\theta}$  and  $\tau$  being constants), by Theorem 2.3, the prediction error bound in the RHS of (11) is  $O(n^{-c / p})$  for some constant  $c > 0^3$ . Since the exponent is dependent on  $p$ , the curse of dimensionality becomes evident. This seems to indicate that the analysis of Proposition 3.1 could potentially be refined. However, given the negative results demonstrated by [20, 8], it is likely that for tree-based estimators, even with the additive model, the curse of dimensionality cannot be circumvented.
Proposition 3.1 requires each component of the additive function to lie in an LRP class. Actually, this condition can be further relaxed as in the following proposition.
Proposition 3.2 Suppose Assumption 2.1 holds true, and  $f^{*}$  has an additive structure as in (14). Given integer  $r \geq 1$  and constants  $\alpha > 0, \beta \geq 1$ . Suppose for each  $k \in [p]$  there exist  $0 = t_0^{(k)} < t_1^{(k)} < \dots < t_{r-1}^{(k)} < t_r^{(k)} = 1$  with  $t_j^{(k)} - t_{j-1}^{(k)} \geq \alpha / r$  and  $f_k^* \in LRP([t_{j-1}, t_j], \beta)$  for all  $j \in [r]$ . Then Assumption 2.2 is satisfied with parameter  $\lambda = \underline{\theta} / (p\bar{\theta}\tau^2)$  where  $\tau^2 = \max \left\{ \frac{2r\bar{\theta}}{\underline{\theta}}, \frac{r^2}{2\alpha} \right\} \max \{9\beta^2, 32 + \beta^2\}$ .
By Proposition 3.2, an additive function  $f^{*}$  satisfies the SID condition if each component function is a piecewise function, with each piece belonging to an LRP class. Notably, continuity at the joining points of consecutive pieces is not necessary. However, for technical reasons, it is required that each piece has a length that is not excessively small, as determined by the parameter  $\alpha$ . See Appendix B for the proofs of Proposition 3.1 and 3.2.
4 Proof sketch of Theorem 2.3
We provide a sketch of the proof of Theorem 2.3. For  $p \geq 1$  and  $d \geq 1$ , define
$$
\mathcal {A} _ {p, d} := \left\{\prod_ {j = 1} ^ {p} \left[ \ell_ {j}, u _ {j} \right] \in \mathcal {A} \mid \# \{j \in [ p ] \mid \left[ \ell_ {j}, u _ {j} \right] \neq [ 0, 1 ] \} \leq d \right\} \tag {17}
$$
That is, each rectangle in  $\mathcal{A}_{p,d}$  has at most  $d$  dimensions that are not the full interval  $[0,1]$ . When  $p\geq d$ ,  $\mathcal{A}_{p,d}$  contains all the rectangles in  $[0,1]^p$ , i.e.  $\mathcal{A}_{p,d} = \mathcal{A}$ . However, when  $d$  is much smaller than  $p$  (in a high-dimensional setting),  $\mathcal{A}_{p,d}$  contains much fewer rectangles than  $\mathcal{A}$ . Note that for a decision tree with depth  $d$ , each leaf node represents a rectangle in  $\mathcal{A}_{p,d}$ .
The proof of Theorem 2.3 builds upon a few technical lemmas which provide uniform bounds between the empirical and populational quantities. These results may hold their own significance in the study of other partition-based algorithms. For  $\delta \in (0,1)$ , define values
$$
\bar {t} _ {1} (\delta) = \bar {t} _ {1} (\delta , n, d) := \frac {4}{n} \log (2 p ^ {d} (n + 1) ^ {2 d} / \delta)
$$
$$
\bar {t} _ {2} (\delta) = \bar {t} _ {2} (\delta , n, d) := \frac {2 \bar {\theta} e ^ {2} d}{n} \vee \frac {\log \left(p ^ {d} (n + 1) ^ {2 d} / \delta\right)}{n} \tag {18}
$$
$$
\bar {t} (\delta) = \bar {t} (\delta , n, d) := \bar {t} _ {1} (\delta , n, d) \vee \bar {t} _ {2} (\delta , n, d)
$$
where  $\bar{\theta}$  is the constant in Assumption 2.1 (i). Note that we have  $\bar{t} (\delta)\leq O(\frac{d\log(np) + \log(1 / \delta)}{n})$  . We have the following uniform bounds on empirical and populational mean on every rectangle in  $\mathcal{A}_{p,d}$
Lemma 4.1 Suppose Assumption 2.1 holds true. Suppose  $\bar{t}_2(\delta / 12) < 3/4$ . Then with probability at least  $1 - \delta$ , it holds
$$
\sup  _ {A \in \mathcal {A} _ {p, d}} \sqrt {\mathbb {P} (X \in A)} \left| \mathbb {E} \left(f ^ {*} (X) \mid X \in A\right) - \bar {y} _ {\mathcal {I} _ {A}} \right| \leq 2 0 U \sqrt {\bar {t} (\delta / 1 2)} \tag {19}
$$
3The lower bound is based on general additive models which may not satisfy the SID condition. When the SID condition is satisfied, the lower bound may be better.
For a tree-based estimator in practice, the prediction at each leaf node is usually given by the sample average of  $y_{i}$  for all the samples  $i$  routed to that leaf, i.e., the value  $\bar{y}_A$  for a leaf with rectangle  $A$ . Therefore, Lemma 4.1 provides a uniform bound on the error if we replace the prediction at each leaf by the populational conditional mean  $\mathbb{E}(f^{*}(X)|X\in A)$ . It is worth noting that the gap between  $\bar{y}_A$  and  $\mathbb{E}(f^{*}(X)|X\in A)$  is rescaled by the quantity  $\sqrt{\mathbb{P}(X\in A)}$ . Intuitively, when the rectangle  $A$  is small and hence  $\mathbb{P}(X\in A)$  is small, few points are routed to the rectangle  $A$ . As a result, the corresponding gap between  $\bar{y}_A$  and  $\mathbb{E}(f^{*}(X)|X\in A)$  can be large - which leads to a large discrepancy of the estimate and the signal locally in this cell  $A$ . However, in another perspective, since the cell  $A$  is small, a large discrepancy in  $A$  only makes a small contribution to the overall error  $\| \widehat{f}^{(d)} - f^{*}\|_{L^2 (X)}^2$ . Therefore, the factor  $\sqrt{\mathbb{P}(X\in A)}$  in the LHS of (19) is a proper balance of these two aspects. This is the key technical argument that differs from the analysis in [10]. It helps establish the tighter bounds in Theorem 2.3.
Lemma 4.2 Suppose Assumption 2.1 holds true. Given a constant  $\alpha >0$ . Given any  $\delta \in (0,1)$ , suppose  $\bar{t}_2(\delta /36) < 3 / 4$ . Then with probability at least  $1 - \delta$ , it holds
$$
\Delta (A, j, b) \leq (1 + \alpha) \widehat {\Delta} (A, j, b) + (1 + 1 / \alpha) \cdot C ^ {\prime} U ^ {2} \bar {t} (\delta / 3 6) \quad \forall A \in \mathcal {A} _ {p, d - 1}, j \in [ p ], b \in \mathbb {R}
$$
$$
\widehat {\Delta} (A, j, b) \leq (1 + \alpha) \Delta (A, j, b) + (1 + 1 / \alpha) \cdot C ^ {\prime} U ^ {2} \bar {t} (\delta / 3 6) \quad \forall A \in \mathcal {A} _ {p, d - 1}, j \in [ p ], b \in \mathbb {R}
$$
for some universal constant  $C' > 0$ .
Lemma 4.2 establishes upper bounds on the discrepancy between empirical and populational impurity decrease. It builds an avenue to translate the SID condition into an empirical counterpart, which is further used to derive the decrease of objective value.
With Lemmas 4.1 and 4.2 at hand, we are ready to wrap up the proof of Theorem 2.3. First, we have
$$
\left\| \widehat {f} ^ {(k)} - f ^ {*} \right\| _ {L ^ {2} (X)} ^ {2} \leq 2 \| f ^ {*} - \widetilde {f} ^ {(k)} \| _ {L ^ {2} (X)} ^ {2} + 2 \| \widetilde {f} ^ {(k)} - \widehat {f} ^ {(k)} \| _ {L ^ {2} (X)} ^ {2} := 2 J _ {1} (k) + 2 J _ {2} (k) \tag {20}
$$
where  $\widetilde{f}^{(k)}$  is a tree with the same splitting rule as  $\widehat{f}^{(k)}$  but leaf predictions replaced by the populational means (in that leaf). The term  $J_{2}(k)$  can be bounded as
$$
J _ {2} (k) = \sum_ {t \in \mathcal {L} ^ {(k)}} \mathbb {P} (X \in A _ {t}) \left(\mathbb {E} \left(f ^ {*} (X) | X \in A _ {t}, \mathcal {X} _ {1} ^ {n}\right) - \bar {y} _ {\mathcal {I} _ {A _ {t}}}\right) ^ {2} \leq C U ^ {2} 2 ^ {k} \cdot \frac {k \log (n p) + \log (1 / \delta)}{n}
$$
where  $\mathcal{L}^{(k)}$  is the set of leaves of  $\widehat{f}^{(k)}$  at depth  $k$ ;  $A_{t}$  is the corresponding rectangle for a leaf  $t$ ; and  $C$  is a universal constant. The last inequality in (21) is by Lemma 4.1. The term  $J_{1}(k)$  can be written as
$$
J _ {1} (k) = \sum_ {t \in \mathcal {L} ^ {(k)}} \mathbb {P} \left(X \in A _ {t} \mid \mathcal {X} _ {1} ^ {n}\right) \cdot \operatorname {V a r} \left(f ^ {*} (X) \mid X \in A _ {t}, \mathcal {X} _ {1} ^ {n}\right)
$$
Making use of Lemma 4.2 and by some algebra (see Appendix A for details), a recursive inequality can be established:
$$
J _ {1} (k + 1) \leq \left(1 - \frac {\lambda}{(1 + \alpha) ^ {2}}\right) J _ {1} (k) + 2 ^ {k} \cdot \frac {C U ^ {2} (k \log (n p) + \log (1 / \delta))}{n}
$$
The proof of Theorem 2.3 is complete by telescoping the inequality above, and combining the upper bounds for  $J_{1}(k)$  and  $J_{2}(k)$  with (20).
5 Conclusion and discussions
We have proved an error bound on the prediction error of CART for a regression problem when  $f^{*}$  satisfies the SID condition. This error bound cannot be improved by more than a log factor. We have also discussed a few sufficient conditions under which we can show that an additive model satisfies the SID condition.
One possible limitation of this work is that: it seems that the SID coefficients  $\lambda$  derived in Section 3 are not tight. Since  $\lambda$  appeared in the exponent of the rate in Theorem 2.3, a tighter estimate of  $\lambda$  leads to an improved convergence rate in (11). We leave it as a future work for a more precise analysis of the SID condition.
A Proof of Theorem 2.3
We first define some notations in the context of the model (1). For  $p \geq 1$  and  $d \geq 1$ , define
$$
\mathcal {A} _ {p, d} := \left\{\prod_ {j = 1} ^ {p} \left[ \ell_ {j}, u _ {j} \right] \in \mathcal {A} \mid \# \{j \in [ p ] \mid [ \ell_ {j}, u _ {j} ] \neq [ 0, 1 ] \} \leq d \right\} \tag {21}
$$
That is, each rectangle in  $\mathcal{A}_{p,d}$  has at most  $d$  dimensions that are not the full interval [0, 1]. Note that for a decision tree with depth  $d$ , each leave node represents a rectangle in  $\mathcal{A}_{p,d}$ . Furthermore, for  $\delta \in (0,1)$ , define values
$$
\bar {t} _ {2} (\delta) = \bar {t} _ {2} (\delta , n, d) := \frac {2 \bar {\theta} e ^ {2} d}{n} \vee \frac {\log \left(p ^ {d} (n + 1) ^ {2 d} / \delta\right)}{n} \tag {22}
$$
where  $\bar{\theta}$  is the constant in Assumption 2.1 (i). Note that we have  $\bar{t} (\delta)\leq O(d\log (np / \delta) / n)$
For two values  $a, b > 0$ , we write  $a \lesssim b$  if there is a universal constant  $C > 0$  such that  $a \leq Cb$ . We write  $a \lesssim_r b$  if there is a constant  $C_r$  that only depends on  $r$  such that  $a \leq C_r b$ .
A.2 Technical lemmas
Now we can introduce the major technical results to establish the error bound.
Lemma A.1 Suppose Assumption 2.1 holds true. Suppose  $\bar{t}_2(\delta / 12) < 3/4$ . Then with probability at least  $1 - \delta$ , it holds
$$
\sup  _ {A \in \mathcal {A} _ {p, d}} \sqrt {\mathbb {P} (X \in A)} | \mathbb {E} (f ^ {*} (X) | X \in A) - \bar {y} _ {\mathcal {I} _ {A}} | \leq 2 0 U \sqrt {\bar {t} (\delta / 1 2)} \tag {23}
$$
The proof of Lemma A.1 is presented in Section A.4. Note that Lemma A.1 provides a uniform bound on the gap between the populational mean  $\mathbb{E}(f^{*}(X)|X\in A)$  and the sample mean  $\bar{y}_{\mathcal{I}_A}$ . This is used to derive the geometric decrease of the bias, using the SID assumption.
Lemma A.2 Suppose Assumption 2.1 holds true. Given any  $\delta \in (0,1)$ , suppose  $\bar{t}_2(\delta /4) < 3 / 4$ . Then with probability at least  $1 - \delta$  it holds
$$
\sup  _ {A \in \mathcal {A} _ {p, d}} \left| \sqrt {\mathbb {P} (X \in A)} - \sqrt {| \mathcal {I} _ {A} | / n} \right| \leq 5 \sqrt {\bar {t} (\delta / 4)} \tag {24}
$$
The proof of Lemma A.2 is presented in Section A.5. Lemma A.2 provides a uniform deviation gap between the square root of probability and sample frequency over all sets in  $\mathcal{A}_{p,d}$ . Note that this uniform bound is stronger than a result without a square root (which can be obtained easily via Hoeffding's inequality and a union bound), and is useful to prove the final error bound in Theorem 2.3.
For any rectangle  $A\in \mathcal{A}$ $j\in [p]$  and  $b\in \mathbb{R}$  , define
$$
\Delta_ {L} (A, j, b) := \mathbb {P} (X \in A _ {L}) \left(\mathbb {E} (f ^ {*} (X) | X \in A) - \mathbb {E} (f ^ {*} (X) | X \in A _ {L})\right) ^ {2}
$$
$$
\Delta_ {R} (A, j, b) := \mathbb {P} (X \in A _ {R}) \left(\mathbb {E} (f ^ {*} (X) | X \in A) - \mathbb {E} (f ^ {*} (X) | X \in A _ {R})\right) ^ {2}
$$
$$
\widehat {\Delta} _ {L} (A, j, b) := \frac {| \mathcal {I} _ {A _ {L}} |}{n} (\bar {y} \mathcal {I} _ {A _ {L}} - \bar {y} \mathcal {I} _ {A}) ^ {2}
$$
$$
\widehat {\Delta} _ {R} (A, j, b) := \frac {| \mathcal {I} _ {A _ {R}} |}{n} (\bar {y} _ {\mathcal {I} _ {A _ {R}}} - \bar {y} _ {\mathcal {I} _ {A}}) ^ {2}
$$
We have the following identity regarding the impurity decrease of each split.
Lemma A.3 For any rectangle  $A \in \mathcal{A}$ ,  $j \in [p]$  and  $b \in \mathbb{R}$ , it holds
$$
\Delta (A, j, b) = \Delta_ {L} (A, j, b) + \Delta_ {R} (A, j, b) \tag {25}
$$
$$
\widehat {\Delta} (A, j, b) = \widehat {\Delta} _ {L} (A, j, b) + \widehat {\Delta} _ {R} (A, j, b)
$$
Proof. We just present the proof of the second equality. The proof of the first equality can be proved similarly. Note that
$$
\begin{array}{l} \widehat {\Delta} (A, j, b) = \frac {1}{n} \sum_ {i \in \mathcal {I} _ {A}} \left(y _ {i} - \bar {y} _ {\mathcal {I} _ {A}}\right) ^ {2} - \frac {1}{n} \sum_ {i \in \mathcal {I} _ {A _ {L}}} \left(y _ {i} - \bar {y} _ {\mathcal {I} _ {A _ {L}}}\right) ^ {2} - \frac {1}{n} \sum_ {i \in \mathcal {I} _ {A _ {R}}} \left(y _ {i} - \bar {y} _ {\mathcal {I} _ {A _ {R}}}\right) ^ {2} \\ = \frac {1}{n} \sum_ {i \in \mathcal {I} _ {A _ {L}}} \left[ \left(y _ {i} - \bar {y} _ {\mathcal {I} _ {A}}\right) ^ {2} - \left(y _ {i} - \bar {y} _ {\mathcal {I} _ {A _ {L}}}\right) ^ {2} \right] + \frac {1}{n} \sum_ {i \in \mathcal {I} _ {A _ {R}}} \left[ \left(y _ {i} - \bar {y} _ {\mathcal {I} _ {A}}\right) ^ {2} - \left(y _ {i} - \bar {y} _ {\mathcal {I} _ {A _ {R}}}\right) ^ {2} \right] \tag {26} \\ \end{array}
$$
$$
\begin{array}{l} \frac {1}{n} \sum_ {i \in \mathcal {I} _ {A _ {L}}} \left[ \left(y _ {i} - \bar {y} _ {\mathcal {I} _ {A}}\right) ^ {2} - \left(y _ {i} - \bar {y} _ {\mathcal {I} _ {A _ {L}}}\right) ^ {2} \right] \\ = \frac {1}{n} \sum_ {i \in \mathcal {I} _ {A _ {L}}} \left[ \left(y _ {i} - \bar {y} \mathcal {I} _ {A}\right) ^ {2} - \left(y _ {i} - \bar {y} \mathcal {I} _ {A}\right) ^ {2} - 2 \left(y _ {i} - \bar {y} \mathcal {I} _ {A}\right) \left(\bar {y} \mathcal {I} _ {A} - \bar {y} \mathcal {I} _ {A _ {L}}\right) - \left(\bar {y} \mathcal {I} _ {A} - \bar {y} \mathcal {I} _ {A _ {L}}\right) ^ {2} \right] \tag {27} \\ = \frac {\left| \mathcal {I} _ {A _ {L}} \right|}{n} \left(\bar {y} _ {\mathcal {I} _ {A}} - \bar {y} _ {\mathcal {I} _ {A _ {L}}}\right) ^ {2} = \widehat {\Delta} _ {L} (A, j, b) \\ \end{array}
$$
$$
\frac {1}{n} \sum_ {i \in \mathcal {I} _ {A _ {R}}} \left[ \left(y _ {i} - \bar {y} _ {\mathcal {I} _ {A}}\right) ^ {2} - \left(y _ {i} - \bar {y} _ {\mathcal {I} _ {A _ {R}}}\right) ^ {2} \right] = \widehat {\Delta} _ {R} (A, j, b) \tag {28}
$$
The proof is complete by combining (26), (27) and (28).
Lemma A.4 Suppose Assumption 2.1 holds true. Given a constant  $\alpha > 0$ , given any  $\delta \in (0,1)$ , suppose  $\bar{t}_2(\delta /36) < 3 / 4$ . Then with probability at least  $1 - \delta$ , it holds
$$
\Delta (A, j, b) \leq (1 + \alpha) \widehat {\Delta} (A, j, b) + (1 + 1 / \alpha) \cdot 5 0 0 0 U ^ {2} \bar {t} (\delta / 3 6) \quad \forall A \in \mathcal {A} _ {p, d - 1}, j \in [ p ], b \in \mathbb {R} \tag {29}
$$
$$
\widehat {\Delta} (A, j, b) \leq (1 + \alpha) \Delta (A, j, b) + (1 + 1 / \alpha) \cdot 5 0 0 0 U ^ {2} \bar {t} (\delta / 3 6) \quad \forall A \in \mathcal {A} _ {p, d - 1}, j \in [ p ], b \in \mathbb {R} \tag {30}
$$
Proof. For  $A \in \mathcal{A}_{p,d-1}$ ,  $j \in [p]$  and  $a \in \mathbb{R}$ , by Lemma A.3 we have
$$
\begin{array}{l} \Delta (A, j, b) = \Delta_ {L} (A, j, b) + \Delta_ {R} (A, j, b) \\ \widehat {\Delta} (A, j, b) = \widehat {\Delta} _ {L} (A, j, b) + \widehat {\Delta} _ {R} (A, j, b) \tag {31} \\ \end{array}
$$
Define the events  $\mathcal{E}_1$  and  $\mathcal{E}_2$ :
$$
\begin{array}{l} \mathcal{E}_{1}:= \left\{\sup_{A\in \mathcal{A}_{p,d}}\sqrt{\mathbb{P}(X\in A)}\Bigl|\mathbb{E}(f^{*}(X)|X\in A) - \bar{y}_{\mathcal{I}_{A}}\Bigr|\leq 20U\sqrt{\bar{t}(\delta / 36)}\right\} \\ \mathcal{E}_{2}:= \left\{\sup_{A\in \mathcal{A}_{p,d}}\left|\sqrt{\mathbb{P}(X\in A)} -\sqrt{|\mathcal{I}_{A}| / n}\right|\leq 5\sqrt{\bar{t}(\delta / 12)}\right\} \\ \end{array}
$$
Then by Lemmas A.1 and A.2, we have  $\mathbb{P}(\mathcal{E}_i) \geq 1 - \delta / 3$  for  $i = 1, 2$ , so we have  $\mathbb{P}(\cap_{i=1}^{2}\mathcal{E}_i) \geq 1 - \delta$ . Below we prove (29) and (30) conditioned on the events  $\mathcal{E}_1$  and  $\mathcal{E}_2$ .
$$
\begin{array}{l} \sqrt {\Delta_ {L} (A , j , a)} = \sqrt {\mathbb {P} (X \in A _ {L})} \left| \mathbb {E} (f ^ {*} (X) | X \in A) - \mathbb {E} (f ^ {*} (X) | X \in A _ {L}) \right| \\ \leq \sqrt {\mathbb {P} (X \in A _ {L})} | \mathbb {E} (f ^ {*} (X) | X \in A) - \bar {y} _ {\mathcal {I} _ {A}} | + \sqrt {\mathbb {P} (X \in A _ {L})} | \bar {y} _ {\mathcal {I} _ {A}} - \bar {y} _ {\mathcal {I} _ {A _ {L}}} | \tag {32} \\ + \sqrt {\mathbb {P} (X \in A _ {L})} | \bar {y} _ {\mathcal {I} _ {A _ {L}}} - \mathbb {E} (f ^ {*} (X) | X \in A _ {L}) | \\ := J _ {1} + J _ {2} + J _ {3} \\ \end{array}
$$
To bound  $J_{1}$ , we have
$$
J _ {1} \leq \sqrt {\mathbb {P} (X \in A)} | \mathbb {E} (f ^ {*} (X) | X \in A) - \bar {y} _ {\mathcal {T} _ {A}} | \leq 2 0 U \sqrt {\bar {t} (\delta / 3 6)} \tag {33}
$$