SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Given a choice of  $\kappa^{\prime \prime},c^{\prime \prime} > 0$  , consider the event (over the draw of  $D$
$$
\begin{array}{l} E := \{\nexists \mathcal {P} ^ {\prime \prime} \subseteq [ n ] \text {w i t h} | \mathcal {P} ^ {\prime \prime} | \geq \left\lfloor \kappa^ {\prime \prime} \cdot n \right\rfloor \text {t h a t f u l f i l l s (D . 3)} \} \\ = \left\{\exists \tilde {\mathcal {P}} \subseteq [ n ] \text {w i t h} | \tilde {\mathcal {P}} | \geq \lceil (1 - \kappa^ {\prime \prime}) n \rceil \text {s u c h t h a t} | f ^ {*} (\boldsymbol {x} _ {i}) - f (\boldsymbol {x} _ {i}) | <   c ^ {\prime \prime} \quad \forall i \in \tilde {\mathcal {P}} \right\}. \\ \end{array}
$$
With the proper choices of  $c''$  and  $\kappa''$  independent of  $n$  and  $f$ , we will show  $P(E) \leq 3n^{-1}$ . We will find a small  $c'' > 0$  such that if  $f^*$  and  $f$  are closer than  $c''$  on too many training points  $\tilde{\mathcal{P}}$
and  $f$  overfits by at least the fraction  $c_{\mathrm{fit}}$ , the noise variables  $\varepsilon_{i}$  on the complement  $\tilde{\mathcal{P}}^c$  would have to be unreasonably large, contradicting the event  $E_{6i}$  defined below, and implying (D.3) with high probability. We will use the notation  $\| \pmb {f}\|_{\mathcal{P}}^{2}\coloneqq \sum_{i\in \mathcal{P}}f(\pmb{x}_{i})^{2}$  and  $\| \pmb {y}\|_{\mathcal{P}}^{2}\coloneqq \sum_{i\in \mathcal{P}}y_{i}^{2}$ .
Step 2b: Noise bounds.
Lemma D.6 (i) states that there exists a  $\kappa'' > 0$  small enough such that the event (over the draw of  $D$ )
$$
E _ {6 i} := \left\{\forall \mathcal {P} _ {1} \subseteq [ n ] \text {w i t h} | \mathcal {P} _ {1} | \leq \left\lfloor \kappa^ {\prime \prime} \cdot n \right\rfloor \text {i t h o l d s t h a t} \frac {1}{n} \| \boldsymbol {f} ^ {*} - \boldsymbol {y} \| _ {\mathcal {P} _ {1}} ^ {2} = \frac {1}{n} \sum_ {i \in \mathcal {P} _ {1}} \varepsilon_ {i} ^ {2} <   \frac {c _ {\text {f i t}}}{4} \sigma^ {2} \right\},
$$
fulfills, for  $n$  large enough,  $P(E_{6i})\geq 1 - n^{-1}$
Lemma D.6 (ii) implies that there exists a  $c_{\mathrm{lower}} > 0$  such that the event (over the draw of  $D$ )
$$
E _ {6 i i} := \left\{\forall \mathcal {P} _ {2} \text {w i t h} | \mathcal {P} _ {2} | \geq \left\lfloor (1 - \kappa^ {\prime \prime}) n \right\rfloor \text {i t h o l d s t h a t} \frac {1}{n} \| \boldsymbol {f} ^ {*} - \boldsymbol {y} \| _ {\mathcal {P} _ {2}} ^ {2} \geq c _ {\text {l o w e r}} \cdot \sigma^ {2} \right\},
$$
fulfills, for  $n$  large enough,  $P(E_{6ii})\geq 1 - n^{-1}$
Lemma D.5 states that the total amount of noise  $\| \varepsilon \|_{[n]}^2$  concentrates around its mean  $n\sigma^2$ . More precisely, we will use that for any  $c_{\varepsilon} \in (0,1)$  the event (over the draw of  $D$ )
$$
E _ {5} := \left\{\frac {1}{n} \| \boldsymbol {f} ^ {*} - \boldsymbol {y} \| _ {[ n ]} ^ {2} \geq c _ {\varepsilon} \cdot \sigma^ {2} \right\},
$$
fulfills  $P(E_5) \geq 1 - \exp \left(-n \cdot \left(\frac{1 - c_{\varepsilon}}{2}\right)^2\right)$ .
Step 2c: Lower bounding  $\| \varepsilon \|_{\tilde{p}^c}^2$
Given some function  $f \in \mathcal{H}$ , assume in steps 2c and 2d that event  $E$  holds and that  $\tilde{\mathcal{P}} \subseteq [n]$  denotes a subset of the training set that fulfills  $|\tilde{\mathcal{P}}| \geq \lceil (1 - \kappa^{\prime \prime})n \rceil$  and  $|f^{*}(\pmb{x}_{i}) - f(\pmb{x}_{i})| < c^{\prime \prime} \quad \forall i \in \tilde{\mathcal{P}}$ .
In step 2c, assume we choose  $\tilde{c}_{\mathrm{fit}} > 0$  such that  $\tilde{c}_{\mathrm{fit}}\| \boldsymbol{f}^{*} - \boldsymbol{y}\|_{\mathcal{P}}^{2} \leq \| \boldsymbol{f} - \boldsymbol{y}\|_{\mathcal{P}}^{2}$ . Then by the overfitting Assumption (O) it holds that
$$
\frac {1}{n} \left(\tilde {c} _ {\text {f i t}} \| \boldsymbol {f} ^ {*} - \boldsymbol {y} \| _ {\mathcal {P}} ^ {2} + \| \boldsymbol {f} - \boldsymbol {y} \| _ {\mathcal {P} ^ {c}} ^ {2}\right) \leq \frac {1}{n} \left(\| \boldsymbol {f} - \boldsymbol {y} \| _ {\mathcal {P}} ^ {2} + \| \boldsymbol {f} - \boldsymbol {y} \| _ {\mathcal {P} ^ {c}} ^ {2}\right) \leq (1 - c _ {\text {f i t}}) \sigma^ {2}. \tag {D.4}
$$
If we restrict ourselves to event  $E_5$ , dropping the term  $\| \pmb{f} - \pmb{y} \|_{\mathcal{P}^c}^2$  in (D.4), then dividing by  $\tilde{c}_{\mathrm{fit}}$  and subtracting the result from the inequality in the definition of event  $E_5$  yields
$$
\frac {1}{n} \| \boldsymbol {\varepsilon} \| _ {\mathcal {P} ^ {c}} ^ {2} = \frac {1}{n} \| \boldsymbol {f} ^ {*} - \boldsymbol {y} \| _ {\mathcal {P} ^ {c}} ^ {2} \geq c _ {\varepsilon} \sigma^ {2} - \frac {1 - c _ {\text {f i t}}}{\tilde {c} _ {\text {f i t}}} \sigma^ {2}. \tag {D.5}
$$
Step 2d: Choosing the constants.
If we choose  $c_{\varepsilon} \coloneqq 1 - \frac{c_{\mathrm{fit}}}{4}$  and  $\tilde{c}_{\mathrm{fit}} \coloneqq \frac{2 - 2c_{\mathrm{fit}}}{2 - c_{\mathrm{fit}}} \in (0,1)$ , then (D.5) becomes
$$
\frac {1}{n} \| \varepsilon \| _ {\mathcal {P} ^ {c}} ^ {2} \geq \frac {c _ {\text {f i t}}}{4} \sigma^ {2}.
$$
Now it is left to show that the condition  $\tilde{c}_{\mathrm{fit}}\| \pmb {f}^{*} - \pmb {y}\|_{\bar{\mathcal{P}}}^{2}\leq \| \pmb {f} - \pmb {y}\|_{\bar{\mathcal{P}}}^{2}$ , that is required for Step 2c, holds with high probability with our choice of  $\tilde{c}_{\mathrm{fit}}$ .
With some arbitrary but fixed  $\varepsilon_{\mathrm{lower}} \in (0, \sqrt{c_{\mathrm{lower}}})$ , choose  $c'' := (1 - \sqrt{\tilde{c}_{\mathrm{fit}}})(\sqrt{\frac{c_{\mathrm{lower}}}{1 - \kappa''}} - \frac{\varepsilon_{\mathrm{lower}}}{\sqrt{1 - \kappa''}})\sigma$ . Then on event  $E_{6ii}$ , for  $n$  large enough, it holds that
$$
\left(1 - \sqrt {\tilde {c} _ {\mathrm {f i t}}}\right) \frac {1}{\sqrt {n}} \| \boldsymbol {f} ^ {*} - \boldsymbol {y} \| _ {\tilde {\mathcal {P}}} \geq \left(1 - \sqrt {\tilde {c} _ {\mathrm {f i t}}}\right) \sqrt {c _ {\mathrm {l o w e r}}} \sigma \geq \sqrt {1 - \kappa^ {\prime \prime}} \cdot c ^ {\prime \prime} + \frac {c ^ {\prime \prime}}{\sqrt {n}}. \tag {D.6}
$$
By definition of  $\tilde{\mathcal{P}}$ , it holds that
$$
\left\| \boldsymbol {f} - \boldsymbol {f} ^ {*} \right\| _ {\tilde {\mathcal {P}}} ^ {2} = \sum_ {i \in \tilde {\mathcal {P}}} \left(f (\boldsymbol {x} _ {i}) - f ^ {*} (\boldsymbol {x} _ {i})\right) ^ {2} <   \lceil (1 - \kappa^ {\prime \prime}) n \rceil (c ^ {\prime \prime}) ^ {2},
$$
$$
\frac {1}{\sqrt {n}} \| \boldsymbol {f} - \boldsymbol {f} ^ {*} \| _ {\tilde {\mathcal {P}}} <   \sqrt {1 - \kappa^ {\prime \prime}} \cdot c ^ {\prime \prime} + \frac {c ^ {\prime \prime}}{\sqrt {n}}. \tag {D.7}
$$
Now, using the triangle inequality, (D.7) and (D.6) yields the condition required for Step 2c,
$$
\begin{array}{l} \frac {1}{\sqrt {n}} \| \boldsymbol {f} - \boldsymbol {y} \| _ {\tilde {\mathcal {P}}} \\ \geq \frac {1}{\sqrt {n}} \| \boldsymbol {f} ^ {*} - \boldsymbol {y} \| _ {\tilde {\mathcal {P}}} - \frac {1}{\sqrt {n}} \| \boldsymbol {f} - \boldsymbol {f} ^ {*} \| _ {\tilde {\mathcal {P}}} \\ \geq \frac {1}{\sqrt {n}} \| \boldsymbol {f} ^ {*} - \boldsymbol {y} \| _ {\tilde {\mathcal {P}}} - \sqrt {1 - \kappa^ {\prime \prime}} \cdot c ^ {\prime \prime} - \frac {c ^ {\prime \prime}}{\sqrt {n}} \\ \geq \sqrt {\tilde {c} _ {\mathrm {f i t}}} \frac {1}{\sqrt {n}} \| \boldsymbol {f} ^ {*} - \boldsymbol {y} \| _ {\hat {\mathcal {P}}}. \\ \end{array}
$$
Step 2e: Upper bounding the probability of  $E$ .
To conclude, we have seen in steps 2c and 2d that on  $E \cap E_{6ii} \cap E_5$ , it holds that
$$
\frac {1}{n} \| \boldsymbol {\varepsilon} \| _ {\mathcal {P} _ {c}} ^ {2} \geq \frac {c _ {\mathrm {f i t}}}{4} \sigma^ {2}.
$$
On  $E_{6i}$ , it holds that
$$
\frac {1}{n} \| \varepsilon \| _ {\bar {P} ^ {c}} ^ {2} <   \frac {c _ {\text {f i t}}}{4} \sigma^ {2}.
$$
Hence  $E_{6i} \cap E \cap E_{6ii} \cap E_5 = \emptyset$ . This implies  $E \subseteq (E_5 \cap E_{6i} \cap E_{6ii})^c$ , where the right hand side is independent of  $f \in \mathcal{H}$  and just depends on the training data  $D$ . Since  $P(E_{6i}) \geq 1 - n^{-1}$  and  $P(E_{6ii} \cap E_5) \geq 1 - n^{-1} - \exp\left(-n \cdot \left(\frac{1 - c_\varepsilon}{2}\right)^2\right)$ , it must hold that, for  $n$  large enough,
$$
P (E) \leq P \left(\left(E _ {5} \cap E _ {6 i} \cap E _ {6 i i}\right) ^ {c}\right) \leq 2 n ^ {- 1} + \exp \left(- n \cdot \left(\frac {1 - c _ {\varepsilon}}{2}\right) ^ {2}\right) \leq 3 n ^ {- 1}.
$$
The following proposition generalizes the inconsistency result for small bandwidths, Proposition 5 in Buchholz (2022), beyond interpolating estimators to estimators whose RKHS norm is at most a constant factor larger than the RKHS norm of the minimum-norm interpolant. The intuition is that if the bandwidth is too small, then the minimum-norm interpolant  $g_{D,\gamma}$  returns to 0 between the training points. Then  $\| g_{D,\gamma}\|_{L_2(\rho)}$  is smaller and bounded away from  $\| f^{*}\|_{L_2(\rho)}$ . We can replace  $g_{D,\gamma}$  by any other function  $f\in \mathcal{H}$  that fulfills Assumption (N).
Proposition D.4 (Inconsistency for small bandwidths). Under the assumptions of Proposition D.2, there exist constants  $B, c > 0$  such that, with probability  $1 - O(n^{-1})$  over the draw of  $D$ : For any function  $f \in \mathcal{H}$  that fulfills Assumption (N) but not necessarily Assumption (O), the excess risk satisfies
where  $c$  depends neither on  $n$  nor on  $\gamma < B n^{-1 / d}$ .
Proof. Denote the upper bound on the Lebesgue density of  $P_{X}$  by  $C_u$ . The triangle inequality implies
$$
\begin{array}{l} \left\| f ^ {*} - f \right\| _ {L _ {2} \left(P _ {X}\right)} \geq \left\| f ^ {*} \right\| _ {L _ {2} \left(P _ {X}\right)} - \left\| f \right\| _ {L _ {2} \left(P _ {X}\right)} \geq \left\| f ^ {*} \right\| _ {L _ {2} \left(P _ {X}\right)} - \sqrt {C _ {u}} \| f \| _ {2} \\ \geq \left\| f ^ {*} \right\| _ {L _ {2} (P _ {X})} - \sqrt {C _ {u}} \left\| f \right\| _ {\mathcal {H}} \geq \left\| f ^ {*} \right\| _ {L _ {2} (P _ {X})} - C _ {\mathcal {H}} \sqrt {C _ {u}} \left\| g _ {D, \gamma} \right\| _ {\mathcal {H}}, \\ \end{array}
$$
where  $\| f\| _2\leq \| f\|_{\mathcal{H}}$  follows from the fact that the Fourier transform  $\hat{k}$  of the kernel satisfies  $\hat{k} (\xi)\leq 1$  Now in the proof of Lemma 17 in Buchholz (2022)  $a > 0$  can be chosen smaller to generalize the statement to
$$
\| g _ {D, \gamma} \| _ {\mathcal {H}} ^ {2} \leq \frac {1}{6 C _ {\mathcal {H}} ^ {2} C _ {u}} \| f ^ {*} \| _ {L _ {2} (P _ {X})} ^ {2} + c _ {9} (\gamma^ {2} n ^ {2 / d} + \gamma^ {2 s} n ^ {2 s / d}),
$$
where  $c_{9}$  depends on  $c_{u},f^{*},d,s$  and  $C_\mathrm{norm}$ . Finally we can choose  $B$  small enough such that Eq. (32) in Buchholz (2022) can be replaced by  $C_{\mathcal{H}}\sqrt{C_u}\| g_{D,\gamma}\|_{\mathcal{H}}\leq \frac{2}{3}\| f^{*}\|_{L_2(P_X)}$  so that we get
$$
\left\| f ^ {*} - f \right\| _ {L _ {2} \left(P _ {X}\right)} \geq \frac {1}{3} \left\| f ^ {*} \right\| _ {L _ {2} \left(P _ {X}\right)} > 0.
$$
D.1 Auxiliary results for the proof of Theorem 1
Lemma D.5 (Concentration of  $\chi_{n}^{2}$  variables). Let  $U$  be a chi-squared distributed random variable with  $n$  degrees of freedom. Then, for any  $c\in (0,1)$  it holds that
$$
P \left(\frac {U}{n} \leq c\right) \leq \exp \left(- n \cdot \left(\frac {1 - c}{2}\right) ^ {2}\right).
$$
Proof. Lemma 1 in Laurent and Massart (2000) implies for any  $x > 0$
$$
P \left(\frac {U}{n} \leq 1 - 2 \sqrt {\frac {x}{n}}\right) \leq \exp (- x).
$$
Solving  $c = 1 - 2\sqrt{\frac{x}{n}}$  for  $x$  yields  $x = n \cdot \left(\frac{1 - c}{2}\right)^2$ .
Lemma D.6. Let  $\varepsilon_1, \ldots, \varepsilon_n$  be i.i.d.  $\mathcal{N}(0, \sigma^2)$  random variables,  $\sigma^2 > 0$ . Let  $(\varepsilon^2)^{(i)}$  denote the  $i$ -th largest of  $\varepsilon_1^2, \ldots, \varepsilon_n^2$ .
(i) A constant fraction of noise cannot concentrate on less than  $\Theta(n)$  points: For all constants  $\alpha, c > 0$  there exists a constant  $C \in (0,1)$  such that with probability at least  $1 - n^{-\alpha}$ , for  $n$  large enough,
$$
\frac {1}{n} \sum_ {i = 1} ^ {\lfloor C n \rfloor} (\varepsilon^ {2}) ^ {(i)} <   c \sigma^ {2}.
$$
(ii)  $\Theta (n)$  points amount to a constant fraction of noise: For all constants  $\alpha >0$  and  $\kappa \in (0,1)$  there exists a constant  $c > 0$  such that with probability at least  $1 - n^{-\alpha}$ , for  $n$  large enough,
$$
\frac {1}{n} \sum_ {i = 1} ^ {\lfloor (1 - \kappa) n \rfloor} \left(\varepsilon^ {2}\right) ^ {(n - i + 1)} \geq c \sigma^ {2}.
$$
Proof. Without loss of generality, we can assume  $\sigma^2 = 1$ .
(i) For a constant  $C\in (0,1)$  yet to be chosen, consider the sum
$$
S _ {C, n} := \frac {1}{n} \sum_ {i = 1} ^ {\lfloor C n \rfloor} \left(\varepsilon^ {2}\right) ^ {(i)}.
$$
For  $T > 0$  yet to be chosen, we consider the random set  $\mathcal{I}_T \coloneqq \{i \in [n] \mid \varepsilon_i^2 > T\}$  and denote its size by  $K \coloneqq |\mathcal{I}_T|$ . To bound  $K$ , we note that  $K = \xi_1 + \ldots + \xi_n$ , where  $\xi_i = \mathbb{1}_{\varepsilon_i^2 > T}$ . We first want to bound  $p_T \coloneqq \mathbb{E}\xi_i = P(\varepsilon_i^2 > T)$ .
The random variables  $\varepsilon_{i}^{2}$  follow a  $\chi_1^2$ -distribution, whose CDF we denote by  $F(t)$  and whose PDF is
$$
f (t) = \mathbb {1} _ {(0, \infty)} (t) C _ {1} t ^ {- 1 / 2} \exp (- t / 2) \tag {D.8}
$$
for some absolute constant  $C_1$ . Moreover, we use  $\Phi$  and  $\phi$  to denote the CDF and PDF of  $\mathcal{N}(0,1)$ , respectively.
Step 1: Tail bounds. Following Duembgen (2010), we have for  $x > 0$ :
$$
1 - \Phi (x) > \frac {2 \phi (x)}{\sqrt {4 + x ^ {2}} + x} \geq \frac {2 \phi (x)}{2 + x + x} = \frac {\phi (x)}{1 + x}
$$
$$
1 - \Phi (x) <   \frac {2 \phi (x)}{\sqrt {2 + x ^ {2}} + x} \leq \frac {2 \phi (x)}{1 + x}.
$$
Hence, for  $t > 0$ , we have
$$
1 - F (t) = 2 \left(1 - \Phi (\sqrt {t})\right) > \frac {2 \phi (\sqrt {t})}{1 + \sqrt {t}} = \sqrt {\frac {2}{\pi}} \frac {\exp (- t / 2)}{1 + \sqrt {t}}
$$
$$
1 - F (t) = 2 (1 - \Phi (\sqrt {t})) <   \frac {4 \phi (\sqrt {t})}{1 + \sqrt {t}} = \sqrt {\frac {8}{\pi}} \frac {\exp (- t / 2)}{1 + \sqrt {t}}.
$$
By choosing  $T \coloneqq -2\log (C\sqrt{\pi / 32}) > 0$ , we obtain
$$
p _ {T} = 1 - F (T) <   \sqrt {\frac {8}{\pi}} \exp (- T / 2) = C / 2.
$$
Step 2: Bounding  $K$ . The random variables  $\xi_{i}$  from above satisfy  $\xi_{i} \in [0,1]$ . By Hoeffding's inequality (Steinwart and Christmann, 2008, Theorem 6.10), we have for  $\tau > 0$
$$
P \left(\frac {1}{n} \sum_ {i = 1} ^ {n} \left(\xi_ {i} - \mathbb {E} \xi_ {i}\right) \geq (1 - 0) \sqrt {\frac {\tau}{2 n}}\right) \leq \exp (- \tau).
$$
We choose  $\tau \coloneqq C^2 n / 2$ , such that with probability  $\geq 1 - \exp (-C^2 n / 2)$ , we have
$$
K / n - p _ {T} = \frac {1}{n} \sum_ {i = 1} ^ {n} (\xi_ {i} - \mathbb {E} \xi_ {i}) \leq \sqrt {\frac {C ^ {2} n / 2}{2 n}} = C / 2.
$$
Suppose that this holds. Then,  $K \leq np_T + Cn / 2 < Cn$  and, since  $K$  is an integer,  $K \leq \lfloor Cn \rfloor$ . This implies
$$
S _ {C, n} \leq \frac {1}{n} \left(\sum_ {i = 1} ^ {K} \left(\varepsilon^ {2}\right) ^ {(i)} + (\lfloor C n \rfloor - K) T\right) \leq C T + \frac {1}{n} \sum_ {i = 1} ^ {K} \left(\varepsilon^ {2}\right) ^ {(i)}. \tag {D.9}
$$
We now want to bound  $\sum_{i=1}^{K} (\varepsilon^2)^{(i)}$ . To this end, we note that conditioned on  $K = k$  for some  $k \in [n]$ , the  $k$  random variables  $(\varepsilon_i)_{i \in \mathcal{I}_T}$  are i.i.d. drawn from the distribution of  $\varepsilon^2$  given  $\varepsilon^2 > T$ , for  $\varepsilon \sim \mathcal{N}(0,1)$ . By  $X, X_1, X_2, \ldots$ , we denote i.i.d. random variables drawn from the distribution of  $\varepsilon^2 - T \mid \varepsilon^2 > T$ . This means that conditioned on  $K = k$ ,
$$
\sum_ {i = 1} ^ {k} \left(\varepsilon^ {2}\right) ^ {(i)} = \sum_ {i \in \mathcal {I} _ {T}} \varepsilon_ {i} ^ {2} \quad \text {i s d i s t r i b u t e d a s} \quad k T + \sum_ {i = 1} ^ {k} X _ {i}. \tag {D.10}
$$
Step 3: Conditional expectation. The density of  $X$  is given by
$$
\begin{array}{l} p _ {X} (t) = \mathbb {1} _ {t > 0} \frac {f (T + t)}{1 - F (T)} \stackrel {\mathrm {(D . 8)}} {\leq} \mathbb {1} _ {t > 0} \frac {C _ {1} (T + t) ^ {- 1 / 2} \exp (- (t + T) / 2)}{\sqrt {2 / \pi} \exp (- T / 2) / (1 + \sqrt {T})} \\ \leq \mathbb {1} _ {t > 0} C _ {2} \exp (- t / 2), \\ \end{array}
$$
where we have used that for  $t > 0$
$$
\frac {1 + \sqrt {T}}{\sqrt {T + t}} \leq \frac {1 + \sqrt {T}}{\sqrt {T}} = 1 + \frac {1}{\sqrt {T}} \leq 2
$$
since  $T = -2\log (C\sqrt{\pi / 32}) \geq -2\log (\sqrt{\pi / 32}) \approx 1.008$ . We can now bound
$$
\begin{array}{l} \mathbb {E} [ X ] = \int_ {0} ^ {\infty} t p _ {X} (t) \mathrm {d} t \\ \leq \int_ {0} ^ {\infty} C _ {2} t \exp (- t / 2) \mathrm {d} t = 4 C _ {2}. \tag {D.11} \\ \end{array}
$$
Step 4: Conditional subgaussian norm. For  $t \geq 0$
$$
\begin{array}{l} P (| X | > t) = P (X > t) = \frac {1 - F (T + t)}{1 - F (T)} \leq 2 \frac {1 + \sqrt {T}}{1 + \sqrt {T + t}} \frac {\exp (- (T + t) / 2)}{\exp (- T / 2)} \\ \leq 2 \exp (- t / 2). \\ \end{array}
$$
Since the denominator 2 in  $2\exp (-t / 2)$  is constant, by Proposition 2.7.1 and Definition 2.7.5 in Vershynin (2018), the subexponential norm  $\| X\|_{\psi_1}$  is therefore bounded by an absolute constant  $C_3$ . Moreover, by Exercise 2.7.10 in Vershynin (2018), we have  $\| X - \mathbb{E}X\|_{\psi_1}\leq C_4\| X\|_{\psi_1}\leq C_5$  for absolute constants  $C_4,C_5$ .
Step 5: Conditional Concentration. Now, Bernstein's inequality for subexponential random variables (Vershynin, 2018, Corollary 2.8.1) yields for  $t \geq 0$  and some absolute constant  $C_6 > 0$ :
$$
P \left(\left| \sum_ {i = 1} ^ {k} X _ {i} - \mathbb {E} X _ {i} \right| \geq t\right) \leq 2 \exp \left(- C _ {6} \min  \left(\frac {t ^ {2}}{k C _ {5} ^ {2}}, \frac {t}{C _ {5}}\right)\right). \tag {D.12}
$$
We choose  $t = C_5Cn$  and obtain for  $k \leq Cn$
$$
\begin{array}{l} P \left(\sum_ {i = 1} ^ {k} \left(\varepsilon^ {2}\right) ^ {(i)} \geq k T + 4 C _ {2} k + C _ {5} C n \mid K = k\right) \\ \stackrel {\text {(D . 1 0)}} {=} P \left(\sum_ {i = 1} ^ {k} X _ {i} \geq 4 C _ {2} k + C _ {5} C n \mid K = k\right) \\ \stackrel {\text {(D . 1 1)}} {\leq} P \left(\left| \sum_ {i = 1} ^ {k} X _ {i} - \mathbb {E} X _ {i} \right| \geq t\right) \\ \stackrel {(D. 1 2)} {\leq} 2 \exp \left(- C _ {6} C n\right). \\ \end{array}
$$
Step 6: Final bound. From Step 2, we know that  $K \leq \lfloor Cn \rfloor$  with probability  $\geq 1 - \exp(-C^2 n / 2)$ . Moreover, in this case, Step 5 yields
$$
\sum_ {i = 1} ^ {K} \left(\varepsilon^ {2}\right) ^ {(i)} <   K T + 4 C _ {2} K + C _ {5} C n \leq C n \left(T + 4 C _ {2} + C _ {5}\right)
$$
with probability  $\geq 1 - \exp (-C_6Cn)$ . By Eq. (D.9), we therefore have
$$
S _ {C, n} <   C T + C \left(T + 4 C _ {2} + C _ {5}\right) = - 4 C \log \left(C \sqrt {\pi / 3 2}\right) + C _ {7} C.
$$
Since  $\lim_{C\searrow 0} - C\log (C) = 0$ , we can choose  $C\in (0,1)$  such that  $-4C\log (C\sqrt{\pi / 32}) + C_7C < c$  for the given constant  $c > 0$  from the theorem statement, and obtain the desired bound with high probability in  $n$ .
(ii) Since the  $\varepsilon_i^2$  are non-negative and their distribution has a density, there must exist  $T > 0$  with  $P(\varepsilon_i^2 < T) \leq (1 - \kappa)/4$ . Similar to the proof of (i), we then want to bound  $K := |\{i \in [n] | \varepsilon_i^2 < T\}| = \xi_1 + \ldots + \xi_i$  with  $\xi_i = \mathbb{1}_{\varepsilon_i^2 < T}$ . The  $\xi_i \in [0,1]$  are independent with  $\mathbb{E}\xi_i = P(\varepsilon_i^2 < T) \leq (1 - \kappa)/4$ . As in Step 2 of (i), Hoeffding's inequality then yields for  $\tau > 0$ :
We set  $\tau \coloneqq (1 - \kappa)^2 n / 2$ , such that with probability  $\geq 1 - \exp ((1 - \kappa)^2 n / 2)$ , we have
$$
\begin{array}{l} K / n - (1 - \kappa) / 4 \leq K / n - P (\varepsilon_ {i} ^ {2} <   T) = \frac {1}{n} \sum_ {i = 1} ^ {n} (\xi_ {i} - \mathbb {E} \xi_ {i}) <   \sqrt {\frac {(1 - \kappa) ^ {2} n / 2}{2 n}} \\ = \frac {1 - \kappa}{2} . \\ \end{array}
$$
In this case, we have
$$
\begin{array}{l} \frac {1}{n} \sum_ {i = 1} ^ {\lfloor (1 - \kappa) n \rfloor} (\varepsilon^ {2}) ^ {(n - i + 1)} \geq \frac {1}{n} (\lfloor (1 - \kappa) n \rfloor - K) T \geq \frac {1}{n} (((1 - \kappa) n - 1) - K) T \\ \geq \left(\frac {1 - \kappa}{4} - \frac {1}{n}\right) T, \\ \end{array}
$$
where the right-hand side is lower bounded by  $c \coloneqq (1 - \kappa)T / 8$  for  $n$  large enough.
The next lemma is a generalization of Lemma 9 in Buchholz (2022) to arbitrary fractions  $\kappa$  of the training points. Therefore, for any  $\kappa \in (0,1)$  define
$$
\delta_ {\min } (\kappa) = n ^ {- 1 / d} \left(\frac {\kappa}{C _ {\rho} \omega_ {d}}\right) ^ {1 / d},
$$
Lemma D.7 (Generalization of Lemma 9 in Buchholz (2022)). Let  $\kappa, \nu \in (0,1)$ , and let  $c_{\Omega} > 0$  be a constant that satisfies  $P_X(\mathrm{dist}(\pmb{x}, \partial \Omega) < c_{\Omega}) \leq \kappa$ . Let  $\mathcal{P} = \{\pmb{x}_1, \dots, \pmb{x}_n\}$  be i.i.d. points distributed according to the measure  $P_X$ , which has lower and upper bounded density on its entire
bounded open Lipschitz domain  $\Omega \subseteq \mathbb{R}^d$ ,  $C_l \leq p_X(\pmb{x}) \leq C_u$ . Then there exists a constant  $\Theta > 0$  depending on  $d, C_u, \nu$  such that with probability at least  $1 - \exp\left(-\frac{3\kappa n}{7}\right)$  there exists a good subset  $\mathcal{P}' \subseteq \mathcal{P}$ ,  $|\mathcal{P}'| \geq (1 - 7\kappa)n$ , with the following properties: For  $\pmb{x} \in \mathcal{P}'$  we have  $\mathrm{dist}(\pmb{x}, \partial \Omega) \geq c_{\Omega}$ ,  $|\pmb{x} - \pmb{y}| > \delta_{\min}(\kappa)$  for  $\pmb{x} \neq \pmb{y} \in \mathcal{P}'$ , and for all  $\pmb{x} \in \mathcal{P}'$  we have
$$
\sum_ {\boldsymbol {y} \in \mathcal {P} ^ {\prime} \backslash \{\boldsymbol {x} \}} | \boldsymbol {x} - \boldsymbol {y} | ^ {- d - 2 \nu} \leq \frac {2 \Theta \delta_ {\min} (\kappa) ^ {- 2 \nu} n}{\kappa^ {2}}.
$$
Proof. First by the definition of  $\delta_{\mathrm{min}}$ , it holds that
$$
P \left(\boldsymbol {x} _ {j} \in \bigcup_ {i <   j} B \left(\boldsymbol {x} _ {i}, \delta_ {\min }\right)\right) \leq C _ {u} \omega_ {d} \delta_ {\min } ^ {d} n \leq \kappa
$$
Also for all  $\pmb{y} \in \Omega$
$$
\begin{array}{l} \mathbb {E} _ {\boldsymbol {x}} \left(\left(\boldsymbol {x} - \boldsymbol {y}\right) ^ {- d - 2 s} \mathbf {1} \left(\left| \boldsymbol {x} - \boldsymbol {y} \right| \geq \delta_ {\min }\right)\right) = \int_ {B \left(\boldsymbol {y}, \delta_ {\min }\right) ^ {c}} | \boldsymbol {x} - \boldsymbol {y} | ^ {- d - 2 \nu} p _ {X} (\boldsymbol {x}) \mathrm {d} \boldsymbol {x} \\ \leq C _ {u} \int_ {B (\boldsymbol {x}, \delta_ {\min }) ^ {c}} | \boldsymbol {x} - \boldsymbol {y} | ^ {- d - 2 \nu} \mathrm {d} \boldsymbol {y} \leq \Theta \delta_ {\min } ^ {- 2 \nu} \\ \end{array}
$$
for some  $\Theta > 0$  depending only on  $C_u$ ,  $d$  and  $\nu$ . We conclude that for each  $j$
$$
P \left(\sum_ {i <   j} | \boldsymbol {x} _ {i} - \boldsymbol {x} _ {j} | ^ {- d - 2 \nu} \mathbf {1} \left(| \boldsymbol {x} _ {i} - \boldsymbol {x} _ {j} | > \delta_ {\min }\right) > \frac {\Theta \delta_ {\min } ^ {- 2 \nu} n}{\kappa}\right) \leq \kappa .
$$
Also  $P$  (dist  $(\pmb{x}_j,\partial \Omega) < c_{\Omega}) < \kappa$ . The union bound implies that
$$
\begin{array}{l} P \left(\boldsymbol {x} _ {j} \notin \bigcup_ {i <   j} B \left(\boldsymbol {x} _ {i}, \delta_ {\min }\right), \sum_ {i <   j} \left| \boldsymbol {x} _ {i} - \boldsymbol {x} _ {j} \right| ^ {- d - 2 \nu} \mathbf {1} _ {\left| \boldsymbol {x} _ {i} - \boldsymbol {x} _ {j} \right| > \delta_ {\min }} <   \frac {\Theta \delta_ {\min } ^ {- 2 \nu} n}{\kappa}, \operatorname {d i s t} \left(\boldsymbol {x} _ {j}, \partial \Omega\right) > c _ {\Omega}\right) \\ = P \left(\boldsymbol {x} _ {j} \notin \bigcup_ {i <   j} B \left(\boldsymbol {x} _ {i}, \delta_ {\min }\right), \sum_ {i <   j} | \boldsymbol {x} _ {i} - \boldsymbol {x} _ {j} | ^ {- d - 2 \nu} <   \frac {\Theta \delta_ {\min } ^ {- 2 \nu} n}{\kappa}, \operatorname {d i s t} \left(\boldsymbol {x} _ {j}, \partial \Omega\right) > c _ {\Omega}\right) \geq 1 - 3 \kappa . \\ \end{array}
$$
We use a martingale construction similar to the one in Lemma 7 of Buchholz (2022) by defining