SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} 0 <   M (\widehat {\theta}) - M \left(\theta_ {0}\right) \\ = M (\widehat {\theta}) - M _ {\beta_ {n}} (\widehat {\theta}) - M \left(\theta_ {0}\right) + M _ {\beta_ {n}} \left(\theta_ {0}\right) \\ + M _ {\beta_ {n}} (\widehat {\theta}) - M _ {\beta_ {n}} (\theta_ {0}) - 2 M _ {n, \beta_ {n}} (\widehat {\theta}) + 2 M _ {n, \beta_ {n}} (\theta_ {0}) \\ + 2 M _ {n, \beta_ {n}} (\widehat {\theta}) - 2 M _ {n} (\widehat {\theta}) - 2 M _ {n, \beta_ {n}} (\theta_ {0}) + 2 M _ {n} (\theta_ {0}) \\ + 2 M _ {n} (\widehat {\theta}) - 2 M _ {n} \left(\theta_ {0}\right) \\ =: S _ {1, n} + S _ {2, n} + S _ {3, n} + S _ {4, n}. \\ \end{array}
$$
By the definition of  $\widehat{\theta}$ , we have  $S_{4,n} \leq 0$ . In following steps, we will bound  $S_{1,n}, S_{2,n}$  and  $S_{3,n}$ , respectively. The truncation level  $\beta_{n}$  is chosen as  $\beta_{n} = \beta_{0} \log(n)$  for  $\beta_{0} \geq 2\sigma_{Y}$ .
Step 2 (Bounding  $S_{1,n}$ ).
For any  $\theta$ , it holds that
$$
\begin{array}{l} M ^ {i} (\theta) - M _ {\beta_ {n}} ^ {i} (\theta) = \mathbb {E} _ {T} \left\{\left(\tilde {Y} - \widehat {\tau} _ {i} (\theta)\right) ^ {2} - \left(T _ {\beta_ {n}} \tilde {Y} - T _ {\beta_ {n}} \widehat {\tau} _ {i} (\theta)\right) ^ {2} I \{X \in Q _ {i} (\theta) \} \right\} \\ = \mathbb {E} _ {T} \left\{\left(\tilde {Y} - T _ {\beta_ {n}} \tilde {Y}\right) \left(\tilde {Y} + T _ {\beta_ {n}} \tilde {Y} - 2 \widehat {\pi} _ {i} (\theta)\right) I \{X \in Q _ {i} (\theta) \} \right\} \\ + \mathbb {E} _ {T} \left\{\left(T _ {\beta_ {n}} \widehat {\pi} _ {i} (\theta) - \widehat {\pi} _ {i} (\theta)\right) \left(T _ {\beta_ {n}} \tilde {Y} + T _ {\beta_ {n}} \widehat {\pi} _ {i} (\theta) - 2 \widehat {\pi} _ {i} (\theta)\right) I \{X \in Q _ {i} (\theta) \} \right\} \\ =: S _ {5, n} + S _ {6, n}. \\ \end{array}
$$
For  $T_{1,n}$ , by Cauchy-Schwarz inequality we have
$$
| S _ {5, n} | \leq \sqrt {\mathbb {E} _ {T} (\tilde {Y} - T _ {\beta_ {n}} \tilde {Y}) ^ {2}} \sqrt {\mathbb {E} _ {T} (\tilde {Y} + T _ {\beta_ {n}} \tilde {Y} - 2 \widehat {\pi} _ {i} (\theta)) ^ {2}} \lesssim \sqrt {\mathbb {E} _ {T} (\tilde {Y} - T _ {\beta_ {n}} \tilde {Y}) ^ {2}},
$$
where the second inequality is because  $\mathbb{E}_T(\tilde{Y}^2) \leq \infty$  and  $\mathbb{E}_T\{\widehat{\pi}_i^2 (\theta)\} \leq \mathbb{E}_T(\tilde{Y}^2) / |Q_i(\theta)|$ . Since
$$
I (| \tilde {Y} | > \beta_ {n}) \leq \frac {\exp (\sigma_ {Y} | Y | ^ {2} / 2)}{\sigma_ {Y} \beta_ {n} ^ {2} / 2},
$$
$$
| T _ {1, n} | \lesssim \sqrt {\mathbb {E} _ {T} (\tilde {Y} - T _ {\beta_ {n}} \tilde {Y}) ^ {2}} \leq \sqrt {\mathbb {E} _ {T} \left\{| Y | ^ {2} \frac {\exp (\sigma_ {Y} | Y | ^ {2} / 2)}{\sigma_ {Y} \beta_ {n} ^ {2} / 2} \right\}} \leq \sqrt {\frac {2}{\sigma_ {Y}} \mathbb {E} _ {T} \exp (\sigma_ {Y} | Y | ^ {2})} \exp (- \frac {\sigma_ {Y} \beta_ {n} ^ {2}}{4}).
$$
Since  $\mathbb{E}_T\exp (\sigma_Y|Y|^2) < \infty$  and  $\beta_{n} = \beta_{0}\log (n)$ , we conclude that  $|S_{5,n}|\lesssim \frac{1}{n}$ . With the same argument, we have  $S_{6,n}\lesssim \frac{1}{n}$ , implying that
$$
M (\theta) - M _ {\beta_ {n}} (\theta) \lesssim \frac {1}{n} \tag {15}
$$
for any  $\theta \in \Theta$ . Therefore, the truncation error  $S_{1,n} \lesssim \frac{1}{n}$ .
Step 3 (Bounding  $S_{2,n}$ ).
Let  $M_{N,of} = \left\{f = (T_{\beta_n}\tilde{Y} - T_{\beta_n}\pi)I(X \in Q_i(\theta)): \theta = (q,c) \in [p] \times \mathbb{R}\right\}$ . By applying Lemma 2 we obtain
$$
\mathcal {N} _ {1} (\delta , M _ {N, o f}, z _ {1} ^ {n}) \leq (p n) ^ {2} \left(\frac {c \beta_ {n}}{\delta}\right) ^ {4},
$$
where  $z_1^n$  is any set  $\{z_1, \dots, z_n\} \in [0, 1]^p \times \mathcal{V}$  and  $c > 0$  is a universal constant. It follows from Lemma 1 that
$$
\begin{array}{l} \mathbb {P} _ {T} \left\{\exists \theta \in \Theta : | M _ {\beta_ {n}} (\theta) - M _ {n, \beta_ {n}} (\theta) | \geq \frac {1}{2} (\alpha + \gamma + M _ {\beta_ {n}} (\theta)) \right\} \\ \leq 2 8 (p n) ^ {2} \left(\frac {8 0 c \beta_ {n} ^ {2}}{\gamma}\right) ^ {4} \exp \left(- \frac {\alpha n}{1 2 8 4 \beta_ {n} ^ {4}}\right) \\ \lesssim \exp \left(- \frac {\alpha n}{\beta_ {n} ^ {4}} + \log (p n) - \log (\gamma)\right). \\ \end{array}
$$
Taking  $\gamma = 1 / n$  and  $\alpha = (t + \log (pn))\beta_n^4 /n$  implies that with probability at least  $1 - C_{1}e^{-t}$  for some universal constant  $C_1 > 0$ ,
$$
\forall \theta \in \Theta , | M _ {\beta_ {n}} (\theta) - 2 M _ {n, \beta_ {n}} (\theta) | \lesssim \frac {t + \log (p n) \log^ {4} (n)}{n}. \tag {16}
$$
Therefore, we conclude that with probability at least  $1 - C_1 e^{-t}$ , the stochastic error  $S_{2,n} \lesssim \left\{ t + \log (pn) \log^4 (n) \right\} / n$ .
Step 4 (Bounding  $S_{3,n}$ ). According to (15), we have
$$
\forall \theta \in \Theta : \mathbb {E} _ {T} \left\{M _ {n, \beta_ {n}} (\theta) - M _ {n} (\theta) \right\} \lesssim \frac {1}{n}
$$
Since  $\tilde{Y}$  is sub-Gaussian by assumption, it is straightforward to see that  $(T_{\beta_n}\tilde{Y} - T_{\beta_n}\pi_i(\theta))^2 I(X \in Q_i(\theta))$  and  $(\tilde{Y} - \pi_i(\theta))^2 I(X \in Q_i(\theta))$  are sub-exponential for  $i \in \{L,R\}$ . Suppose  $\left\| (T_{\beta_n}\tilde{Y} - T_{\beta_n}\pi_i(\theta))^2 I(X \in Q_i(\theta))\right\|_{\psi_1} \leq \sigma_0$  and  $\left\| (\tilde{Y} - \pi_i(\theta))^2 I(X \in Q_i(\theta))\right\|_{\psi_1} \leq \sigma_0$  for all  $\theta \in \Theta$ , where  $\|\cdot\|_{\psi_1}$  is the sub-exponential norm operator. By applying Bernstein's inequality, for any  $s > 0$ , we have
$$
\begin{array}{l} \left. \mathbb {P} _ {T} \left\{\left| M _ {n, \beta_ {n}} ^ {i} (\theta) - M _ {n} ^ {i} (\theta) - \mathbb {E} _ {T} \left\{M _ {n, \beta_ {n}} ^ {i} (\theta) - M _ {n} ^ {i} (\theta) \right\} \geq s \right| \right\} \right. \\ \leq 2 \exp \left(- c \min \left(\frac {n s ^ {2}}{\sigma_ {0} ^ {2}}, \frac {n s}{\sigma_ {0}}\right)\right), \\ \end{array}
$$
for  $i \in \{R, L\}$ , where  $c > 0$  is a universal constant. Taking  $s = \frac{\sigma_0 t}{cn} = C_2 t$ , for any  $t \geq 0$  we obtain
$$
\mathbb {P} _ {T} \left\{\left| M _ {n, \beta_ {n}} ^ {i} (\theta) - M _ {n} ^ {i} (\theta) - \mathbb {E} _ {T} \left\{M _ {n, \beta_ {n}} ^ {i} (\theta) - M _ {n} ^ {i} (\theta) \right\} \geq C _ {2} t \right| \right\} \leq 2 \exp (- t) \tag {17}
$$
for any  $n > t / c$ . Since the above result holds for any  $\theta \in \Theta$ , we conclude that for any  $t > 0$ , with probability at least  $1 - 4e^{-t}$ , we have  $S_{3,n} \lesssim (t + 1) / n$ .
Combining the results on  $S_{1,n}, S_{2,n}$  and  $S_{3,n}$ , we conclude that for any  $t > 0$ , with probability at least  $1 - C_3 e^{-t}$ , it holds that
$$
\mathcal {L} _ {i} (\widehat {\theta}, \widehat {\pi} _ {i} (\widehat {\theta})) - \mathcal {L} _ {i} \left(\theta_ {0}, \pi \left(\theta_ {0}\right)\right) \lesssim \frac {t + \log (p n) \log^ {4} (n)}{n}, \tag {18}
$$
for some universal constants  $C_3, C_4 > 0$ .
Part 2. The proposed scale criterion can reformulated as follows. For  $i \in \{L, R\}$ , let
$$
F _ {0, i} (\theta) = \left\{1 - w _ {i} (\theta) \right\} \left(\tilde {Y} _ {0} - \tau_ {0, i} (\theta)\right) ^ {2} I \left(X _ {0} \in Q _ {i} (\theta)\right)
$$
$$
F _ {1, i} (\theta) = w _ {i} (\theta) \left(\tilde {Y} _ {1} - \tau_ {1, i} (\theta)\right) ^ {2} I \left(X _ {1} \in Q _ {i} (\theta)\right) \text {a n d}
$$
where  $\tau_{0,i}(\theta) = \mathbb{E}_n(\tilde{Y}_0|X_0\in Q_i(\theta))$  and  $\tau_{1,i}(\theta) = \mathbb{E}_m(\tilde{Y}_1|X_1\in Q_i(\theta))$ , and
$$
w _ {i} (\theta) = \sigma_ {u} ^ {2} (Q _ {i} (\theta)) / \left\{\sigma_ {u} ^ {2} (Q _ {i} (\theta)) + \sigma_ {b} ^ {2} (Q _ {i} (\theta)) + b ^ {2} (Q _ {i} (\theta)) \right\},
$$
where  $\sigma_u^2(Q_i(\theta)) = \mathrm{Var}_n(\tau_{0,i}(\theta)), \sigma_b^2(Q_i(\theta)) = \mathrm{Var}_m(\tau_{1,i}(\theta))$  and  $b(Q_i(\theta)) = \tau_{1,i}(\theta) - \tau_{0,1}(\theta)$ . Let  $\mathcal{F}_{s,i}(\theta) = \mathbb{E}_s(F_{s,i}(\theta))$  for  $s \in \{0,1\}$  and  $\mathcal{F}_s(\theta) = \mathcal{F}_{s,L}(\theta) + \mathcal{F}_{s,R}(\theta)$ , the population criterion is defined as  $M_{of}(\theta) = \mathcal{F}_0(\theta) + \mathcal{F}_1(\theta)$ . For the empirical criterion, we first define  $\mathcal{F}_{n,i}(\theta) = \mathbb{E}_n(F_i^R(\theta))$  and  $\mathcal{F}_{m,i}(\theta) = \mathbb{E}_m(F_i^R(\theta))$ . Let  $M_{N,of}(\theta) = \mathcal{F}_{n,L}(\theta) + \mathcal{F}_{n,R}(\theta)$  and  $\mathcal{F}_m(\theta) = \mathcal{F}_{m,L}(\theta) + \mathcal{F}_{m,R}(\theta)$ , the empirical criterion is denoted as  $M_{N,of}(\theta) = M_{N,of}(\theta) + \mathcal{F}_m(\theta)$ . The population and empirical optimal splits are defined by
$$
\theta_ {o f} = \underset {\theta \in \Theta} {\arg \min } M _ {o f} (\theta) \text {a n d} \widehat {\theta} _ {f} = \underset {\theta \in \Theta} {\arg \min } M _ {N, o f} (\theta).
$$
We first have the following error decomposition:
$$
\begin{array}{l} M _ {o f} (\widehat {\theta}) - M _ {o f} \left(\theta_ {0}\right) = M _ {o f} (\widehat {\theta}) - M _ {o f, \beta_ {n}} \left(\widehat {\theta} _ {o f}\right) + M _ {o f} \left(\theta_ {0}\right) - M _ {o f, \beta_ {n}} \left(\theta_ {o f}\right) \\ + M _ {o f, \beta_ {n}} (\widehat {\theta} _ {o f}) + M _ {o f, \beta_ {n}} (\theta_ {o f}) - 2 M _ {N, o f, \beta_ {n}} (\widehat {\theta} _ {o f}) + 2 M _ {N, o f, \beta_ {n}} (\theta_ {o f}) \\ + 2 M _ {N, o f, \beta_ {n}} (\widehat {\theta} _ {o f}) - 2 M _ {N, o f} (\widehat {\theta} _ {o f}) - 2 M _ {N, o f, \beta n} (\theta_ {o f}) + 2 M _ {N, o f} (\theta_ {o f}) \\ + 2 M _ {N, o f} \left(\widehat {\theta} _ {o f}\right) - 2 M _ {N, o f} \left(\theta_ {o f}\right) \\ =: T _ {1, n} + T _ {2, n} + T _ {3, n} + T _ {4, n}. \\ \end{array}
$$
By the definition of  $\widehat{\theta}_{of}$ , we have  $T_{4,n} \leq 0$ . In the following steps, we will bound  $T_{1,n}, T_{2,n}$  and  $T_{3,n}$ , respectively. Following the same argument as for  $S_{1,n}$ , it can be obtained that  $T_{1,n} \lesssim \frac{1}{n}$ .
We now bound  $T_{2,n}$ . Let  $\mathcal{G}_n = \{g : g(y,x) = \sqrt{1 - w(\theta)}\tilde{y} - \tau\} I(x \in \mathcal{Q}(\theta), \theta \in \Theta_n)\}$  and  $\mathcal{H}_n = \{h : h(y,x) = \sqrt{w(\theta)}(\tilde{y} - \tau)I(x \in \mathcal{Q}(\theta), \theta \in \Theta_n)\}$ , then via Lemma 2 we have
$$
\mathcal {N} _ {1} (\delta , \mathcal {G} _ {n}, z _ {1} ^ {n}) \leq (p n) ^ {2} \left(\frac {c \beta_ {n}}{\delta}\right) ^ {4} \text {a n d} \mathcal {N} _ {1} (\delta , \mathcal {H} _ {n}, z _ {1} ^ {n}) \leq (p n) ^ {2} \left(\frac {c \beta_ {n}}{\delta}\right) ^ {4},
$$
for any  $\delta > 0$ , where  $z_1^n$  is any set  $\{z_1, \dots, z_n\} \in [0,1]^p \times \mathcal{V}$  and  $c > 0$  is a universal constant. It follows from Lemma 1 that for any  $\alpha_1, \gamma_1 > 0$
$$
\begin{array}{l} \mathbb {P} _ {T} \left\{\exists \theta \in \Theta_ {n}: \left| \mathcal {F} _ {0, \beta_ {n}} (\theta) - \mathcal {F} _ {n, \beta_ {n}} (\theta) \right| \geq \frac {1}{2} \left(\alpha_ {1} + \gamma_ {1} + M _ {o f, \beta_ {n}} (\theta)\right) \right\} \\ \leq 2 8 (p n) ^ {2} \left(\frac {8 0 c \beta_ {n} ^ {2}}{\gamma_ {1}}\right) ^ {4} \exp \left(- \frac {\alpha_ {1} n}{1 2 8 4 \beta_ {n} ^ {4}}\right) \\ \lesssim \exp \left(- \frac {\alpha_ {1} n}{\beta_ {n} ^ {4}} + \log (p n) - \log (\gamma_ {1})\right). \\ \end{array}
$$
Taking  $\gamma_{1} = 1 / n$  and  $\alpha_{1} = (t + \log (pn))\beta_{n}^{4} / n$  implies that with probability at least  $1 - C_4e^{-t}$  for some universal constant  $C_4 > 0$ ,
$$
\forall \theta \in \Theta_ {n}, \left| M _ {o f, \beta_ {n}} (\theta) - 2 \mathcal {F} _ {n, \beta_ {n}} (\theta) \right| \lesssim \frac {t + \log (p n) \log^ {4} (n)}{n}. \tag {19}
$$
Similar, for any  $\alpha_{2},\gamma_{2} > 0$
$$
\begin{array}{l} \mathbb {P} _ {O} \left\{\exists \theta \in \Theta_ {n}: | \mathcal {F} _ {1, \beta_ {n}} (\theta) - \mathcal {F} _ {m, \beta_ {n}} (\theta) | \geq \frac {1}{2} \left(\alpha_ {1} + \gamma_ {1} + \mathcal {F} _ {1, \beta_ {n}} (\theta)\right) \right\} \\ \lesssim \exp \left(- \frac {\alpha_ {1} m}{\beta_ {n} ^ {4}} + \log (p n) - \log (\gamma_ {1})\right). \\ \end{array}
$$
Taking  $\gamma_{2} = 1 / n$  and  $\alpha_{2} = (t + \log (pn))\beta_{n}^{4} / m$  implies that with probability at least  $1 - C_5e^{-t}$  for some universal constant  $C_5 > 0$ ,
$$
\forall \theta \in \Theta_ {n}, \left| \mathcal {F} _ {1, \beta_ {n}} (\theta) - 2 \mathcal {F} _ {m, \beta_ {n}} (\theta) \right| \lesssim \frac {t + \log (p n) \log^ {4} (n)}{m}. \tag {20}
$$
Combining (19) and (20) delivers that with probability at least  $1 - 2C_{1}e^{-t}$ ,
$$
T _ {2, n} \lesssim \frac {\log (p n) \log^ {4} (n)}{m} + \frac {t + \log (p n) \log^ {4} (n)}{n}, \tag {21}
$$
for ant  $t > 0$ , since  $\widehat{\theta}_{of}, \theta_{f} \in \Theta_{n}$  and  $M_{of,\beta_n}(\theta) = \mathcal{F}_{0,\beta_n}(\theta) + \mathcal{F}_{1,\beta_n}(\theta)$  and  $M_{N,off,\beta_n}(\theta) = \mathcal{F}_{n,\beta_n}(\theta) + \mathcal{F}_{m,\beta_n}(\theta)$  for any  $\theta \in \Theta_{n}$ .
Now we turn to  $T_{3,n}$ , the truncation error for the empirical loss. With the similar argument as in (16), we have with probability at least  $1 - 4e^{-t}$ , it holds that  $T_{3,n} \lesssim (t + 1) / n + (t + 1) / m$  for any  $t > 0$ .
Combining the results for  $T_{1,n}, T_{2,n}$  and  $T_{3,n}$ , we conclude that for any  $t > 0$ , with probability at least  $1 - C_6 e^{-t}$ ,
$$
M _ {o f} \left(\widehat {\theta} _ {o f}\right) - M _ {o f} \left(\theta_ {o f}\right) \lesssim \frac {t + \log (p n) \log^ {4} (n)}{m} + \frac {t + \log (p n) \log^ {4} (n)}{n}, \tag {22}
$$
which completes our proof.
D Supporting lemmas
The following to lemmas are from Section 11.3 and Section 13.1 of Györfi et al. (2002), which are useful for our proofs.
Lemma 1. (Deviation inequality of quadratic process). Suppose that  $\mathcal{G}$  is a class of uniformly bounded functions  $\mathcal{G} = \{g:\mathbb{R}^d\to \mathbb{R}\| g\|_\infty \leq M\}$ . Let  $\mathcal{F} = \{g^2:g\in \mathcal{G}\}$ . Then for any  $n\geq 1$ , it holds that
$$
\begin{array}{l} \mathbb {P} \left\{\exists f \in \mathcal {F}: \left| \mathbb {E} \left\{f (z) \right\} - \mathbb {E} _ {n} \left\{f (z) \right\} \right| \geq \varepsilon (\alpha + \gamma) + \mathbb {E} \left\{f (z) \right\} \right\} \\ \leq 28 \sup _ {z _ {1} ^ {n}} \mathcal {N} _ {1} (\frac {\gamma \varepsilon}{2 0 M}, \mathcal {G}, x _ {1} ^ {n}) \exp \left(- \frac {\varepsilon^ {2} (1 - \varepsilon) \alpha n}{2 1 4 (1 + \varepsilon) M ^ {4}}\right), \\ \end{array}
$$
where  $z_1^n = (z_1,\dots ,z_n)\in \mathbb{R}^d$ $\alpha ,\gamma >0$  and  $0 <   \varepsilon \leq 1 / 2$
Lemma 2. (Covering number of piece-wise constant functions). Let  $\Pi$  be the family of partitions of  $[0,1]^p$ . For any set  $x_1^n = \{x_1,\dots ,x_n\} \subset [0,1]^p$ , let  $\Delta (\Pi)$  be the maximal number of partitions of  $x_{1}^{n}$  induced by elements of  $\Pi$ . Let  $M(\Pi)$  be the maximal number of sets contained in a partition  $\mathcal{P}\in \Pi$ . Denote the piece-wise constant functions on  $[0,1]^p$  be  $\mathcal{F}(\Pi)$  with  $\| f\|_{\infty}\leq \beta_{n}$  for any  $f\in \mathcal{F}(\Pi)$ . Then using Lemma 13.31 and Theorem 9.4 of Györfi et al. (2002) we have
$$
\mathcal {N} _ {1} (\delta , \mathcal {F} (\Pi), x _ {1} ^ {n}) \leq \Delta_ {n} (\Pi) \left(\frac {c _ {1} \beta_ {n}}{\delta}\right) ^ {2 M (\Pi)},
$$
for any  $\delta > 0$ , where  $c_{1} > 0$  is some univiscal constant. Specifically, in each partition for a node  $\mathcal{C}_k$  of a tree, we have  $M(\Pi) = 2$  and  $\Delta_n(\Pi) \leq (pa_n)^2$ , where  $a_{n}$  is the sample size of  $\mathcal{C}_k$ .