SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof of Theorem D.4. Define  $R = \max_{m} \sup_{h \in \mathcal{H}^{(m)}} P_n(y - h(\boldsymbol{x}))^2$ . For any  $h \in \mathcal{H}^{(m)}$ , we obtain
$$
P _ {n} (h ^ {(m)} (\boldsymbol {x}) - \hat {h} ^ {(m)} (\boldsymbol {x})) ^ {2} \leq 2 P _ {n} (y - h ^ {(m)} (\boldsymbol {x})) ^ {2} + 2 P _ {n} (y - \hat {h} ^ {(m)} (\boldsymbol {x})) ^ {2} \leq 4 \sup  _ {h \in \mathcal {H} ^ {(m)}} P _ {n} (y - h (\boldsymbol {x})) ^ {2} \leq 4 R.
$$
From the symmetry of the  $\sigma_{i}$  and the fact that  $\mathcal{H}^{(m)}$  is convex and symmetric, we obtain the following:
$$
\begin{array}{l} \hat {\Re} _ {S} \{h \in \mathcal {H}: \max  P _ {n} (h ^ {(m)} - \hat {h} ^ {(m)}) ^ {2} \leq 4 R \} \\ = \mathbb{E}_{\sigma}\sup_{\substack{h^{(m)}\in \mathcal{H}^{(m)}\\ P_{n}(h^{(m)} - \hat{h}^{(m)})^{2}\leq 4R}}\frac{1}{n}\sum_{i = 1}^{n}\sigma_{i}\sum_{m = 1}^{2}h^{(m)}(\boldsymbol{x}_{i}) \\ = \mathbb{E}_{\sigma}\sup_{\substack{h^{(m)}\in \mathcal{H}^{(m)}\\ P_{n}(h^{(m)} - \hat{h}^{(m)})^{2}\leq 4R}}\frac{1}{n}\sum_{i = 1}^{n}\sigma_{i}\sum_{m = 1}^{2}(h^{(m)}(\boldsymbol{x}_{i}) - \hat{h}^{(m)}(\boldsymbol{x}_{i})) \\ \leq \mathbb{E}_{\sigma}\sup_{\substack{h^{(m)},g^{(m)}\in \mathcal{H}^{(m)}\\ P_{n}(h^{(m)} - g^{(m)})^{2}\leq 4R}}\frac{1}{n}\sum_{i = 1}^{n}\sigma_{i}\sum_{m = 1}^{2}(h^{(m)}(\boldsymbol{x}_{i}) - g^{(m)}(\boldsymbol{x}_{i})) \\ = 2\mathbb{E}_{\sigma}\sup_{\substack{h^{(m)}\in \mathcal{H}^{(m)}\\ P_{n}h^{(m)2}\leq R}}\frac{1}{n}\sum_{i = 1}^{n}\sigma_{i}\sum_{m = 1}^{2}h^{(m)}(\boldsymbol{x}_{i}) \\ \leq 2\sum_{m = 1}^{2}\mathbb{E}_{\sigma}\sup_{\substack{h^{(m)}\in \mathcal{H}^{(m)}\\ P_{n}h^{(m)2}\leq R}}\frac{1}{n}\sum_{i = 1}^{n}\sigma_{i}h^{(m)}(\boldsymbol{x}_{i}) \\ \leq 2 \sum_ {m = 1} ^ {2} \left\{\frac {2}{n} \sum_ {j = 1} ^ {n} \min  \{R, \hat {\lambda} _ {j} ^ {(m)} \} \right\} ^ {\frac {1}{2}} \\ \leq \left\{\frac {1 6}{n} \sum_ {m = 1} ^ {2} \sum_ {j = 1} ^ {n} \min  \{R, \hat {\lambda} _ {j} ^ {(m)} \} \right\} ^ {\frac {1}{2}}. \\ \end{array}
$$
The second inequality comes from the subadditivity of supremum and the third inequality follows from Theorem 6.6 of [20]. To obtain the last inequality, we use  $\sqrt{x} + \sqrt{y} \leq \sqrt{2(x + y)}$ . Thus, we have
$$
\begin{array}{l} 2 c _ {1} \hat {\Re} _ {S} \{h \in \mathcal {H}: \max P _ {n} (h ^ {(m)} - \hat {h} ^ {(m)}) ^ {2} \leq 4 R \} + \frac {(c _ {2} + 2) \eta}{n} \\ \leq 4 c _ {1} \left\{\frac {1 6}{n} \sum_ {m = 1} ^ {2} \sum_ {j = 1} ^ {n} \min \left\{R, \hat {\lambda} _ {j} ^ {(m)} \right\} \right\} ^ {\frac {1}{2}} + \frac {(c _ {2} + 2) \eta}{n}, \\ \end{array}
$$
for some constants  $c_{1}$  and  $c_{2}$ . To apply Corollary D.3, we should solve the following inequality for  $r$
$$
r \leq 4 c _ {1} \left\{\frac {1 6}{n} \sum_ {m = 1} ^ {2} \sum_ {j = 1} ^ {n} \min \left\{r, \hat {\lambda} _ {j} ^ {(m)} \right\} \right\} ^ {\frac {1}{2}}.
$$
For any integers  $\kappa_{m}\in [0,n]$ , the right-hand side is bounded as
$$
\begin{array}{l} 4 c _ {1} \left\{\frac {1 6}{n} \sum_ {m = 1} ^ {2} \sum_ {j = 1} ^ {n} \min  \left\{r, \hat {\lambda} _ {j} ^ {(m)} \right\} \right\} ^ {\frac {1}{2}} \leq 4 c _ {1} \left\{\frac {1 6}{n} \sum_ {m = 1} ^ {2} \left(\sum_ {j = 1} ^ {\kappa_ {m}} r + \sum_ {j = \kappa_ {m} + 1} ^ {n} \hat {\lambda} _ {j} ^ {(m)}\right) \right\} ^ {\frac {1}{2}} \\ = \left\{\left(\frac {2 5 6 c _ {1} ^ {2}}{n} \sum_ {m = 1} ^ {2} \kappa_ {m}\right) r + \frac {2 5 6 c _ {1} ^ {2}}{n} \sum_ {m = 1} ^ {2} \sum_ {j = \kappa_ {m} + 1} ^ {n} \hat {\lambda} _ {j} ^ {(m)} \right\} ^ {\frac {1}{2}}, \\ \end{array}
$$
and we obtain the solution  $r^*$  as
$$
\begin{array}{l} r ^ {*} \leq \frac {1 2 8 c _ {1} ^ {2}}{n} \sum_ {m = 1} ^ {2} \kappa_ {m} + \left(\left\{\frac {1 2 8 c _ {1} ^ {2}}{n} \sum_ {m = 1} ^ {2} \kappa_ {m} \right\} ^ {2} + \frac {2 5 6 c _ {1} ^ {2}}{n} \sum_ {m = 1} ^ {2} \sum_ {j = \kappa_ {m} + 1} ^ {n} \hat {\lambda} _ {j} ^ {(m)}\right) ^ {\frac {1}{2}} \\ \leq \frac {2 5 6 c _ {1} ^ {2}}{n} \sum_ {m = 1} ^ {2} \kappa_ {m} + \left(\frac {2 5 6 c _ {1} ^ {2}}{n} \sum_ {m = 1} ^ {2} \sum_ {j = \kappa_ {m} + 1} ^ {n} \hat {\lambda} _ {j} ^ {(m)}\right) ^ {\frac {1}{2}}. \\ \end{array}
$$
Optimizing the right-hand side with respect to  $\kappa_{1}$  and  $\kappa_{2}$ , we obtain the solution as
$$
r ^ {*} \leq \min  _ {0 \leq \kappa_ {1}, \kappa_ {2} \leq n} \left\{\frac {2 5 6 c _ {1} ^ {2}}{n} \sum_ {m = 1} ^ {2} \kappa_ {m} + \left(\frac {2 5 6 c _ {1} ^ {2}}{n} \sum_ {m = 1} ^ {2} \sum_ {j = \kappa_ {m} + 1} ^ {n} \hat {\lambda} _ {j} ^ {(m)}\right) ^ {\frac {1}{2}} \right\}.
$$
Furthermore, according to Corollary D.3, there exists a constant  $c$  such that with probability at least  $1 - 5e^{-\eta}$ ,
$$
\begin{array}{l} P (y - \hat {h} (x)) ^ {2} - P (y - h ^ {*} (x)) ^ {2} \\ \leq c \left(\min  _ {0 \leq \kappa_ {1}, \kappa_ {2} \leq n} \left\{\frac {1}{n} \sum_ {m = 1} ^ {2} \kappa_ {m} + \left(\frac {1}{n} \sum_ {m = 1} ^ {2} \sum_ {j = \kappa_ {m} + 1} ^ {n} \hat {\lambda} _ {j} ^ {(m)}\right) ^ {\frac {1}{2}} \right\} + \frac {\eta}{n}\right). \\ \end{array}
$$
With Theorem D.4 and Assumption 4.3, we prove Theorem D.4 as follows.
Proof of Theorem 4.4. Using the inequality  $\sqrt{x + y} \leq \sqrt{x} + \sqrt{y}$  for  $x \geq 0, y \geq 0$ , we have
$$
\begin{array}{l} P (y - \hat {h} (\boldsymbol {x})) ^ {2} - P (y - h ^ {*} (\boldsymbol {x})) ^ {2} \\ = O \left(\min  _ {0 \leq \kappa_ {1}, \kappa_ {2} \leq n} \left\{\frac {\kappa_ {1} + \kappa_ {2}}{n} + \left(\frac {1}{n} \sum_ {j = \kappa_ {1} + 1} ^ {n} \hat {\lambda} _ {j} ^ {(1)} + \frac {1}{n} \sum_ {j = \kappa_ {2} + 1} ^ {n} \hat {\lambda} _ {j} ^ {(2)}\right) ^ {\frac {1}{2}} \right\} + \frac {\eta}{n}\right) \\ \leq O \left(\min  _ {0 \leq \kappa_ {1}, \kappa_ {2} \leq n} \left\{\frac {\kappa_ {1} + \kappa_ {2}}{n} + \left(\frac {1}{n} \sum_ {j = \kappa_ {1} + 1} ^ {n} j ^ {- \frac {1}{s _ {1}}} + \frac {1}{n} \sum_ {j = \kappa_ {2} + 1} ^ {n} j ^ {- \frac {1}{s _ {2}}}\right) ^ {\frac {1}{2}} \right\} + \frac {\eta}{n}\right) \\ \leq O \left(\min  _ {0 \leq \kappa_ {1}, \kappa_ {2} \leq n} \left\{\frac {\kappa_ {1} + \kappa_ {2}}{n} + \left(\frac {1}{n} \sum_ {j = \kappa_ {1} + 1} ^ {n} j ^ {- \frac {1}{s _ {1}}}\right) ^ {\frac {1}{2}} + \left(\frac {1}{n} \sum_ {j = \kappa_ {2} + 1} ^ {n} j ^ {- \frac {1}{s _ {2}}}\right) ^ {\frac {1}{2}} \right\} + \frac {\eta}{n}\right). \\ \end{array}
$$
Because it holds that
$$
\sum_ {j = \kappa_ {m} + 1} ^ {n} j ^ {- \frac {1}{s _ {m}}} <   \int_ {\kappa_ {m}} ^ {\infty} x ^ {- \frac {1}{s _ {m}}} \mathrm {d} x <   \left[ \frac {1}{1 - \frac {1}{s _ {m}}} x ^ {1 - \frac {1}{s _ {m}}} \right] _ {\kappa_ {m}} ^ {\infty} = \frac {s _ {m}}{1 - s _ {m}} \kappa_ {m} ^ {1 - \frac {1}{s _ {m}}},
$$
for  $m = 1,2$ , we should solve the following minimization problem:
$$
\min  _ {0 \leq \kappa_ {1}, \kappa_ {2} \leq n} \left\{\frac {\kappa_ {1} + \kappa_ {2}}{n} + \left(\frac {1}{n} \frac {s _ {1}}{1 - s _ {1}} \kappa_ {1} ^ {1 - \frac {1}{s _ {1}}}\right) ^ {\frac {1}{2}} + \left(\frac {1}{n} \frac {s _ {1}}{1 - s _ {1}} \kappa_ {2} ^ {1 - \frac {1}{s _ {1}}}\right) ^ {\frac {1}{2}} \right\} \equiv g (\kappa).
$$
Taking the derivative, we have
$$
\frac {\partial g (\kappa)}{\partial \kappa_ {1}} = \frac {1}{n} + \frac {1}{2} \left(\frac {1}{n} \frac {s _ {1}}{1 - s _ {1}} \kappa_ {1} ^ {1 - \frac {1}{s _ {1}}}\right) ^ {- \frac {1}{2}} \left(- \frac {\kappa_ {1} ^ {- \frac {1}{s _ {1}}}}{n}\right).
$$
Setting this to zero, we find the optimal  $\kappa_{1}$  as
$$
\kappa_ {1} = \left(\frac {s _ {1}}{1 - s _ {1}} \frac {4}{n}\right) ^ {\frac {s _ {1}}{1 + s _ {1}}}.
$$
$$
\kappa_ {2} = \left(\frac {s _ {2}}{1 - s _ {2}} \frac {4}{n}\right) ^ {\frac {s _ {2}}{1 + s _ {2}}},
$$
$$
\begin{array}{l} P (y - \hat {h} (\boldsymbol {x})) ^ {2} - P (y - h ^ {*} (\boldsymbol {x})) ^ {2} \\ \leq O \left(\frac {1}{n} \left(\frac {s _ {1}}{1 - s _ {1}} \frac {4}{n}\right) ^ {\frac {s _ {1}}{1 + s _ {1}}} + \frac {1}{n} \left(\frac {s _ {2}}{1 - s _ {2}} \frac {4}{n}\right) ^ {\frac {s _ {2}}{1 + s _ {2}}} \right. \\ + 2 ^ {\frac {1 - s _ {1}}{1 + s _ {1}}} \left(\frac {s _ {1}}{1 - s _ {1}} \frac {1}{n}\right) ^ {\frac {1}{1 + s _ {1}}} + 2 ^ {\frac {1 - s _ {2}}{1 + s _ {2}}} \left(\frac {s _ {2}}{1 - s _ {2}} \frac {1}{n}\right) ^ {\frac {1}{1 + s _ {2}}} + \frac {\eta}{n} \\ = O \left(n ^ {- \frac {1}{1 + s _ {1}}} + n ^ {- \frac {1}{1 + s _ {2}}}\right) \\ = O \left(n ^ {- \frac {1}{1 + \max \{s _ {1} , s _ {2} \}}}\right). \\ \end{array}
$$