SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
E _ {j} := \left\{\boldsymbol {x} _ {j} \in \bigcup_ {i <   j} B \left(\boldsymbol {x} _ {i}, \delta_ {\min }\right), \quad \text {o r} \quad \sum_ {i <   j} | \boldsymbol {x} _ {i} - \boldsymbol {x} _ {j} | ^ {- d - 2 \nu} \geq \frac {\Theta \delta_ {\min } ^ {- 2 \nu} n}{\kappa}, \quad \text {o r} \quad \operatorname {d i s t} (\boldsymbol {x} _ {j}, \partial \Omega) \leq c _ {\Omega} \right\}.
$$
Now define  $S_{n} \coloneqq \sum_{i=1}^{n} \mathbf{1}_{E_{i}}$ . Using the filtration  $\mathcal{F}_{i} = \sigma(\boldsymbol{x}_{1}, \ldots, \boldsymbol{x}_{i})$ ,  $S_{n}$  can be decomposed into  $S_{n} = M_{n} + A_{n}$ , where  $M_{n}$  is a martingale and  $A_{n}$  is predictable with respect to  $\mathcal{F}_{n}$ . We then get  $A_{n} \leq \sum_{i=1}^{n} P(E_{i} | \mathcal{F}_{i-1}) \leq 3\kappa n$  as well as  $\operatorname{Var}(M_{i} | \mathcal{F}_{i-1}) \leq 3\kappa$ . Hence Freedman's inequality Theorem D.8 yields
$$
P (S _ {n} \geq 6 \kappa n) \leq P (A _ {n} \geq 3 \kappa n) + P (M _ {n} \geq 3 \kappa n) \leq \exp \left(- \frac {3 \kappa n}{7}\right).
$$
This implies that with probability at least  $1 - \exp \left(-\frac{3\kappa n}{7}\right)$  we can find a subset  $\mathcal{P}_s = \{\pmb {z}_1,\dots ,\pmb {z}_m\}$  with  $|\mathcal{P}_s|\geq (1 - 6\kappa)n$  on which it holds that  $\min_{i\neq j}|\pmb {z}_i - \pmb {z}_j|\geq \delta_{\mathrm{min}}$ $\mathrm{dist}(\pmb {z}_j,\partial \Omega)\geq c_\Omega$  and
$$
\sum_ {i \neq j} | \pmb {z} _ {i} - \pmb {z} _ {j} | ^ {- d - 2 \nu} \leq \frac {2 \Theta \delta_ {\mathrm {m i n}} ^ {- 2 \nu} n ^ {2}}{\kappa}.
$$
Using Markov's inequality we see that there are at most  $\kappa n$  points in  $\mathcal{P}_s$  such that
$$
\sum_ {\boldsymbol {z} ^ {\prime} \in \mathcal {P} _ {s}, \boldsymbol {z} \neq \boldsymbol {z} ^ {\prime}} | \boldsymbol {z} - \boldsymbol {z} ^ {\prime} | ^ {- d - 2 \nu} \geq \frac {2 \Theta \delta_ {\min} ^ {- 2 \nu} n}{\kappa^ {2}}.
$$
Removing those points we find a subset  $\mathcal{P}' \subset \mathcal{P}_s$  such that  $|\mathcal{P}'| \geq (1 - 7\kappa)n$  and for each  $z \in \mathcal{P}'$
$$
\sum_ {\boldsymbol {z} ^ {\prime} \in \mathcal {P} _ {s}, \boldsymbol {z} \neq \boldsymbol {z} ^ {\prime}} | \boldsymbol {z} - \boldsymbol {z} ^ {\prime} | ^ {- d - 2 \nu} \leq \frac {2 \Theta \delta_ {\mathrm {m i n}} ^ {- 2 \nu} n}{\kappa^ {2}}.
$$
Theorem D.8 (Freedman's inequality, Theorem 6.1 in Chung and Lu (2006)). Let  $M_{i}$  be a discrete martingale adapted to the filtration  $\mathcal{F}_i$  with  $M_0 = 0$  that satisfies for all  $i \geq 0$
$$
\left| M _ {i + 1} - M _ {i} \right| \leq K
$$
$$
\operatorname {V a r} \left(M _ {i} \mid \mathcal {F} _ {i - 1}\right) \leq \sigma_ {i} ^ {2}.
$$
$$
P \left(M _ {n} - \mathbb {E} \left(M _ {n}\right) \geq \lambda\right) \leq e ^ {- \frac {\lambda^ {2}}{2 \sum_ {i = 1} ^ {n} \sigma_ {i} ^ {2} + K \lambda / 3}}.
$$
E Translating between  $\mathbb{R}^d$  and  $\mathbb{S}^d$
Since the RKHS of the ReLU NTK and NNGP kernels mentioned in Theorem 4 are equivalent to the Sobolev spaces  $H^{(d + 1) / 2}(\mathbb{S}^d)$  and  $H^{(d + 3) / 2}(\mathbb{S}^d)$ , respectively (Chen and Xu, 2021, Bietti and Bach, 2021) (detailed summary in Appendix B.4). Inconsistency of functions in these RKHS that fulfill Assumptions (O) and (N), as in Theorem 1, follows immediately by adapting Theorem 1 via Lemma E.1. In particular, inconsistency holds for the gradient flow and gradient descent estimators  $f_{t,\rho}$  and  $f_{t,\rho}^{\mathrm{GD}}$  as soon as they overfit with lower bounded probability.
For arbitrary open sphere caps  $T \coloneqq \{ \pmb{x} \in \mathbb{S}^d \mid x_{d+1} < v \}, v \in (-1,1)$ , and the open unit ball  $B_1(0) \coloneqq \{ \pmb{y} \in \mathbb{R}^d \mid \| \pmb{y} \|_2 < 1 \}$ , define the scaled stereographic projection  $\phi : T \to B_1(0) \subseteq \mathbb{R}^d$  as
$$
\phi (x _ {1}, \dots , x _ {d + 1}) = \left(\frac {c _ {v} x _ {1}}{1 - x _ {d + 1}}, \dots , \frac {c _ {v} x _ {d}}{1 - x _ {d + 1}}\right),
$$
where the normalization constant  $c_{v} = \sqrt{\frac{1 - v}{1 + v}}$  ensures surjectivity.
Straightforward calculations show that  $\phi$  defines a diffeomorphism. Its inverse  $\phi^{-1}: B_1(0) \to T$  is given by
$$
\phi^ {- 1} (y _ {1}, \ldots , y _ {d}) = \left(\frac {2 c _ {v} ^ {- 1} y _ {1}}{c _ {v} ^ {- 2} \| \boldsymbol {y} \| _ {2} ^ {2} + 1}, \ldots , \frac {2 c _ {v} ^ {- 1} y _ {d}}{c _ {v} ^ {- 2} \| \boldsymbol {y} \| _ {2} ^ {2} + 1}, \frac {c _ {v} ^ {- 2} \| \boldsymbol {y} \| _ {2} ^ {2} - 1}{c _ {v} ^ {- 2} \| \boldsymbol {y} \| _ {2} ^ {2} + 1}\right).
$$
We can translate kernel learning with the kernel  $k$  on  $\mathbb{S}^d$  and the probability distribution  $P$ , where  $P_X$  is supported on  $T$ , to kernel learning with a transformed kernel  $\tilde{k}$  and  $\tilde{P}$  using a sufficiently smooth diffeomorphism like  $\phi : T \to B_1(0) \subseteq \mathbb{R}^d$ . If the RKHS of  $k$  is equivalent to  $H^s(\mathbb{S}^d)$  then the RKHS of  $\tilde{k}$  is equivalent to  $H^s(B_1(0))$ . We formalize this argument in the following lemma. As a consequence it suffices to prove all inconsistency results for Sobolev kernels on  $B_1(0)$ .
Lemma E.1 (Transfer to sphere caps). Let  $k$  be a kernel on  $\mathbb{S}^d$  whose RKHS is equivalent to a Sobolev space  $H^{s}(\mathbb{S}^{d})$ . For fixed  $v \in (-1,1)$ , consider an "open sphere cap"  $T := \{ \pmb{x} \in \mathbb{S}^d \mid x_{d+1} < v \}$ . Furthermore, consider a distribution  $P$  such that  $P_X$  is supported on  $T$  and has lower and upper bounded density  $p_X$  on  $T$ , i.e.,  $0 < C_l \leq p_X(\pmb{x}) \leq C_u < \infty$  for all  $\pmb{x} \in T$ . Then
-  $\tilde{k}(\pmb{x},\pmb{x}^{\prime}) := k(\phi^{-1}(\pmb{x}),\phi^{-1}(\pmb{x}^{\prime}))$  defines a positive definite kernel on  $B_{1}(0) \subseteq \mathbb{R}^{d}$  whose RKHS is equivalent to the Sobolev space  $H^{s}(B_{1}(0))$ ,
-  $\tilde{P} := P \circ \psi^{-1}$  with  $\psi(\boldsymbol{x}, y) := (\phi(\boldsymbol{x}), y)$  defines a probability distribution such that  $\tilde{P}_{\tilde{X}}$  has lower and upper bounded density on  $B_{1}(0) \subseteq \mathbb{R}^{d}$ ,
and kernel learning with  $(k, P)$  or with  $(\tilde{k}, \tilde{P})$  is equivalent in the following sense:
For every function  $f \in \mathcal{H}(k|_T)$  the transformed function  $\tilde{f} = f \circ \phi^{-1} \in \mathcal{H}(\tilde{k})$  has the same RKHS norm, i.e.  $\| f \|_{\mathcal{H}(k|_T)} = \| \tilde{f} \|_{\mathcal{H}(\tilde{k})}$ . Furthermore, the excess risks of  $f$  over  $P$  and of  $\tilde{f}$  over  $\tilde{P}$  coincide, i.e.
$$
\mathbb {E} _ {\boldsymbol {x} \sim P _ {X}} (f (\boldsymbol {x}) - f _ {P} ^ {*} (\boldsymbol {x})) ^ {2} = \mathbb {E} _ {\tilde {\boldsymbol {x}} \sim \tilde {P} _ {X}} (\tilde {f} (\tilde {\boldsymbol {x}}) - \tilde {f} _ {\tilde {P}} ^ {*} (\tilde {\boldsymbol {x}})) ^ {2},
$$
where  $\tilde{f}_{\tilde{P}}^{*}(\tilde{\boldsymbol{x}}) = \mathbb{E}_{(\tilde{X},\tilde{Y})\sim \tilde{P}}(\tilde{Y} |\tilde{X} = \tilde{\boldsymbol{x}})$  denotes the Bayes optimal predictor under  $\tilde{P}$ .
Remark E.2. Many kernel regression estimators can be explicitly written as  $f_{D}^{k}(\pmb{x}) = \hat{f}_{n}(k(\pmb{x},\pmb{X}),k(\pmb{X},\pmb{X}),\pmb{y})$  where  $\hat{f}_n:\mathbb{R}^n\times \mathbb{R}^{n\times n}\times \mathbb{R}^n\to \mathbb{R}$  denotes a measurable function for all  $n\in \mathbb{N}$ . Then the explicit form is preserved under the transformation, i.e.  $f\circ \phi^{-1} = f_{\tilde{D}}^{\tilde{k}}$  with the transformed data set  $\tilde{D} = \{(\phi (\pmb {x}_i),y_i)\}_{i\in [n]}$ .
Proof of Lemma E.1. Step 1: Bounded density. For  $i \in [d], j \in [d + 1]$ , the partial derivatives of  $\phi$  are given by
$$
\partial_ {x _ {j}} \phi_ {i} (\boldsymbol {x}) = \left\{ \begin{array}{l l} \frac {c _ {v}}{1 - x _ {d + 1}}, & \text {f o r} i = j, \\ \frac {c _ {v} x _ {i}}{(1 - x _ {d + 1}) ^ {2}}, & \text {f o r} i \in [ d ], j = d + 1, \\ 0, & \text {o t h e r w i s e}. \end{array} \right.
$$
Given an arbitrary multi-index  $\alpha$ , the partial derivatives  $\partial_{\alpha}\phi_{i} \in L^{2}(T)$ ,  $\partial_{\alpha}\phi_{j}^{-1} \in L^{2}(B_{1}(0))$  are bounded for all  $i \in [d]$ ,  $j \in [d + 1]$ , using  $x_{d + 1} \leq v < 1$  and the inverse function theorem.
Now define  $\tilde{k}(\boldsymbol{x},\boldsymbol{x}^{\prime}) \coloneqq k(\phi^{-1}(\boldsymbol{x}),\phi^{-1}(\boldsymbol{x}^{\prime}))$ ,  $\psi(\boldsymbol{x},y) \coloneqq (\phi(\boldsymbol{x}),y)$  and  $\tilde{P} \coloneqq P \circ \psi^{-1}$ . Then using integration by substitution (Stroock et al., 2011, Theorem 5.2.16), the Lebesgue density of  $\tilde{P}_X$  is given by
$$
p _ {\tilde {X}} (\tilde {\boldsymbol {x}}) = p _ {X} \left(\phi^ {- 1} (\tilde {\boldsymbol {x}})\right) J \phi^ {- 1} (\tilde {\boldsymbol {x}}),
$$
$$
J \phi^ {- 1} (\tilde {\boldsymbol {x}}) := \left[ \det  \left(\left(\left\langle \partial_ {i} \phi^ {- 1} (\tilde {\boldsymbol {x}}), \partial_ {j} \phi^ {- 1} (\tilde {\boldsymbol {x}}) \right\rangle_ {\mathbb {R} ^ {d + 1}}\right) _ {i, j \in \{1, \dots , d \}}\right) \right] ^ {1 / 2}.
$$
$J\phi$  and  $J\phi^{-1}$  can be continuously extended to  $\bar{T}$  and  $\bar{B}_1(0)$ , respectively. Then, since  $J\phi^{-1}$  is continuous on a compact set and because  $\phi$  with the extended domain remains a diffeomorphism so that  $J\phi^{-1}$  cannot attain the value 0, there exists a constant  $C_{\phi} > 0$  such that  $\frac{1}{C_{\phi}} \leq J\phi^{-1}(\tilde{\boldsymbol{x}}) \leq C_{\phi}$  for all  $\tilde{\boldsymbol{x}} \in B_1(0)$ . Hence,  $p_{\tilde{X}}$  is lower and upper bounded.
Step 2: Excess risks coincide. If  $(\tilde{X},\tilde{Y})\sim \tilde{P}$ , the Bayes predictor of  $\tilde{Y}$  given  $\tilde{X}$  is given by  $\tilde{f}^{*}(\tilde{\boldsymbol{x}}) = \mathbb{E}(\tilde{Y} |\tilde{X} = \tilde{\boldsymbol{x}}) = f^{*}(\phi^{-1}(\tilde{\boldsymbol{x}}))$
Let  $\pi_1(\pmb{x},y) = \pmb{x}$  be the projection onto the first component. Then,  $\phi (\pi_1(\pmb {x},y)) = \phi (\pmb {x}) =$ $\pi_1(\phi (\pmb {x}),y) = \pi_1(\psi (\pmb {x},y))$  and hence
$$
\begin{array}{l} \mathbb {E} _ {\boldsymbol {x} \sim P _ {X}} \left(f (\boldsymbol {x}) - f ^ {*} (\boldsymbol {x})\right) ^ {2} = \mathbb {E} _ {(\boldsymbol {x}, y) \sim P} \left(f \left(\pi_ {1} (\boldsymbol {x}, y)\right) - f ^ {*} \left(\pi_ {1} (\boldsymbol {x}, y)\right)\right) ^ {2} \\ = \mathbb {E} _ {(\boldsymbol {x}, \boldsymbol {y}) \sim P} \left(f \left(\phi^ {- 1} \left(\phi \left(\pi_ {1} (\boldsymbol {x}, \boldsymbol {y})\right)\right)\right) - f ^ {*} \left(\phi^ {- 1} \left(\phi \left(\pi_ {1} (\boldsymbol {x}, \boldsymbol {y})\right)\right)\right) ^ {2} \right. \\ = \mathbb {E} _ {(\boldsymbol {x}, \boldsymbol {y}) \sim P} \left(\tilde {f} \left(\pi_ {1} (\psi (\boldsymbol {x}, \boldsymbol {y}))\right) - \tilde {f} ^ {*} \left(\pi_ {1} (\psi (\boldsymbol {x}, \boldsymbol {y}))\right)\right) ^ {2} \\ = \mathbb {E} _ {(\boldsymbol {x}, y) \sim \tilde {P}} \left(\tilde {f} \left(\pi_ {1} (\boldsymbol {x}, y)\right) - \tilde {f} ^ {*} \left(\pi_ {1} (\boldsymbol {x}, y)\right)\right) ^ {2} \\ = \mathbb {E} _ {\boldsymbol {x} \sim \tilde {P} _ {\hat {X}}} \left(\tilde {f} (\boldsymbol {x}) - \tilde {f} ^ {*} (\boldsymbol {x})\right) ^ {2}. \\ \end{array}
$$
Step 3: Transformed RKHS. We want to show that  $\mathcal{H}(k|_T) \to \mathcal{H}(\tilde{k}), f \mapsto f \circ \phi^{-1}$  defines an isometric isomorphism, which especially shows the statement  $\|f\|_{\mathcal{H}(k|_T)} = \|f\|_{\mathcal{H}(\tilde{k})}$  from the proposition. For this, we use the following theorem characterizing RKHSs:
Theorem E.3 (Theorem 4.21 in Steinwart and Christmann (2008)). Let  $k: X \times X \to \mathbb{R}$  be a positive definite kernel function with feature space  $H_0$  and feature map  $\Phi_0: X \to H_0$ . Then
$$
H = \left\{f: X \rightarrow \mathbb {R} \mid \exists w \in H _ {0}: f = \langle w, \Phi_ {0} (\cdot) \rangle_ {H _ {0}} \right\} w i t h
$$
$$
\| f \| _ {H} := \inf  \left\{\| w \| _ {H _ {0}}: f = \langle w, \Phi_ {0} (\cdot) \rangle_ {H _ {0}} \right\},
$$
is the only RKHS for which  $k$  is a reproducing kernel.
A feature map for  $k|_T$  is given by  $\Phi : T \to \mathcal{H}(k|_T)$ ,  $\Phi(\boldsymbol{x}) = k(\boldsymbol{x}, \cdot)$ . Hence a feature map for  $\tilde{k}$  is given by  $\Phi \circ \phi^{-1} : \bar{B}_1(0) \to \mathcal{H}(k|_T)$ . Theorem E.3 states that
$$
\begin{array}{l} \left. \mathcal {H} (k | _ {T}) = \left\{f: T \rightarrow \mathbb {R} \mid \exists w \in \mathcal {H} (k | _ {T}): f = \langle w, \Phi (\cdot) \rangle_ {\mathcal {H} (k | _ {T})} \right\} \text {w i t h} \right. \tag {E.1} \\ \| f \| _ {\mathcal {H} (k | _ {T})} := \inf  \{\| w \| _ {\mathcal {H} (k | _ {T})}: f = \langle w, \Phi (\cdot) \rangle_ {\mathcal {H} (k | _ {T})} \}, \\ \end{array}
$$
$$
\begin{array}{l} \mathcal {H} (\tilde {k}) = \left\{\tilde {f}: B _ {1} (0) \rightarrow \mathbb {R} \mid \exists w \in \mathcal {H} (k | _ {T}): \tilde {f} = \langle w, \Phi \circ \phi^ {- 1} (\cdot) \rangle_ {\mathcal {H} (k | _ {T})} \right\} \text {w i t h} \tag {E.2} \\ \| \tilde {f} \| _ {\mathcal {H} (\tilde {k})} := \inf  \{\| w \| _ {\mathcal {H} (k | _ {T})}: \tilde {f} = \langle w, \Phi \circ \phi^ {- 1} (\cdot) \rangle_ {\mathcal {H} (k | _ {T})} \}. \\ \end{array}
$$
As  $\phi^{-1}$  is bijective, this characterization induces an isometric isomorphism between  $\mathcal{H}(k|_T)$  and  $\mathcal{H}(\tilde{k})$  by mapping  $f = \langle w,\Phi (\cdot)\rangle_{\mathcal{H}(k|_T)}\in \mathcal{H}(k|_T)$  to  $\tilde{f} = f\circ \phi^{-1} = \langle w,\Phi \circ \phi^{-1}(\cdot)\rangle_{\mathcal{H}(k|_T)}\in \mathcal{H}(\tilde{k})$
This shows  $\| f\|_{\mathcal{H}(k|_T)} = \| \tilde{f}\|_{\mathcal{H}(\tilde{k})}$
Step 4: RKHS of  $\tilde{k}$ . We now show that the RKHS of  $\tilde{k}$ , denoted as  $\mathcal{H}(\tilde{k})$ , is equivalent to  $H^{s}(B_{1}(0))$ . To this end, denoting  $\mathcal{A} \circ \phi \coloneqq \{f \circ \phi \mid f \in \mathcal{A}\}$  and  $\mathcal{A}|_{T} \coloneqq \{f|_{T} \mid f \in \mathcal{A}\}$ , we show the following equality of sets (ignoring the norms):
$$
\mathcal {H} (\tilde {k}) \circ \phi \stackrel {{(\mathrm {I})}} {{=}} \mathcal {H} (k | _ {T}) \stackrel {{(\mathrm {I I})}} {{=}} \mathcal {H} (k) | _ {T} \stackrel {{(\mathrm {I I I})}} {{=}} H ^ {s} (\mathbb {S} ^ {d}) | _ {T} \stackrel {{(\mathrm {I V})}} {{=}} H ^ {s} (B _ {1} (0)) \circ \phi .
$$
Since  $\phi$  is bijective, this implies  $\mathcal{H}(\tilde{k}) = H^s(B_1(0))$  as sets, and the norm equivalence then follows from Lemma F.7.
Equality (I) follows from Step 3. Equality (II) follows from Theorem E.3 by observing that if  $\Phi$  is a feature map for  $k$ , then  $\Phi|_T$  is a feature map for  $k|_T$ . Equality (III) holds by assumption. To show (IV), we need a characterization of  $H^s(\mathbb{S}^d)$  that allows to work with charts like  $\phi$ .
Step 4.1: Chart-based characterization of  $H^{s}(\mathbb{S}^{d})$ . A trivialization of a Riemannian manifold  $(M,g)$  with bounded geometry of dimension  $d$  consists of a locally finite open covering  $\{U_{\alpha}\}_{\alpha \in I}$  of  $M$ , smooth diffeomorphisms  $\kappa_{\alpha}:V_{\alpha}\subset \mathbb{R}^{d}\to U_{\alpha}$ , also called charts, and a partition of unity  $\{h_\alpha \}_{\alpha \in I}$  of  $M$  that fulfills  $\mathrm{supp}(h_\alpha)\subseteq U_\alpha$ ,  $0\leq h_{\alpha}\leq 1$  and  $\sum_{\alpha \in I}h_{\alpha} = 1$ . An admissible trivialization of  $(M,g)$  is a uniformly locally finite trivialization of  $M$  that is compatible with geodesic coordinates, for details see (Schneider and GroBe, 2013, Definition 12).
In our case, define an open neighborhood of  $T$  by  $U_{1} \coloneqq \{x \in \mathbb{S}^{d} \mid x_{d+1} < v + \varepsilon\}$  with some  $\varepsilon \in (0, 1 - v)$  arbitrary but fixed, and  $U_{2} \coloneqq \{x \in \mathbb{S}^{d} \mid x_{d+1} > v + \varepsilon/2\}$ . It holds that  $U_{1} \cup U_{2} = \mathbb{S}^{d}$ . Moreover, there exists an appropriate partition of unity consisting of  $C^{\infty}$  functions  $h_{1}, h_{2}: \mathbb{S}^{d} \to [0, 1]$ . Especially, we have  $h_{1}(T) \subseteq h_{1}(U_{2}^{c}) = \{1\}$ . Let  $\phi_{1}: U_{1} \to B_{r_{1}}(0)$  denote the stereographic projection with respect to  $x_{0} = (0, \ldots, 0, 1)$  as above, scaled such that  $\phi_{1}|_{T} = \phi$  and hence  $\phi_{1}(T) = B_{1}(0)$ . Similarly, let  $\phi_{2}: U_{2} \to B_{r_{2}}(0)$  denote an arbitrarily scaled stereographic projection with respect to  $x_{0} = (0, \ldots, 0, -1)$ . Then  $(\{U_{1}, U_{2}\}, \{\phi_{1}^{-1}, \phi_{1}^{-1}\}, \{h_{1}, h_{2}\})$  yields an admissible trivialization of  $\mathbb{S}^{d}$  consisting of only two charts. A detailed derivation can be found in (Hubbert et al., 2015, Section 1.7). Therefore (Schneider and Große, 2013, Theorem 14) lets us define the Sobolev norm on  $\mathbb{S}^{d}$  (up to equivalence) as<sup>2</sup>
$$
\begin{array}{l} \left\| g \right\| _ {H ^ {s} (\mathbb {S} ^ {d})} := \left(\sum_ {\alpha \in I} \left\| \left(h _ {\alpha} g\right) \circ \kappa_ {\alpha} \right\| _ {H ^ {s} (\mathbb {R} ^ {d})} ^ {2}\right) ^ {1 / 2} \\ = \left(\| (h _ {1} g) \circ \phi_ {1} ^ {- 1} \| _ {H ^ {s} (\mathbb {R} ^ {d})} ^ {2} + \| (h _ {2} g) \circ \phi_ {2} ^ {- 1} \| _ {H ^ {s} (\mathbb {R} ^ {d})} ^ {2}\right) ^ {1 / 2}, \\ \end{array}
$$
for any distribution  $g \in \mathcal{D}'(\mathbb{S}^d)$  (i.e. any continuous linear functional on  $C_c^\infty(\mathbb{S}^d)$ ). Then  $g \in H^s(\mathbb{S}^d)$  if and only if  $\| g\|_{H^s(\mathbb{S}^d)} < \infty$ .
Step 4.2: Showing (IV). First, let  $g \in H^{s}(\mathbb{S}^{d})$ . Then, as we saw in Step 4.1, we must have  $\| (h_{1}g) \circ \phi_{1}^{-1} \|_{H^{s}(\mathbb{R}^{d})} < \infty$  and thus  $(h_{1}g) \circ \phi_{1}^{-1} \in H^{s}(\mathbb{R}^{d})$ . By our discussion in Appendix B.1, we then have
$$
(g | _ {T}) \circ \phi^ {- 1} = ((h _ {1} g) \circ \phi_ {1} ^ {- 1}) | _ {B _ {1} (0)} \in H ^ {s} (B _ {1} (0)),
$$
which shows  $g|_{T}\in H^{s}(B_{1}(0))\circ \phi$
Now, let  $f \in H^{s}(B_{1}(0))$ . Then, again following our discussion in Appendix B.1, there exists an extension  $\bar{f} \in H^{s}(\mathbb{R}^{d})$  with  $\bar{f}|_{B_1(0)} = f$ . The set  $\mathcal{B} \coloneqq \phi_1(U_1 \setminus U_2)$  is a closed ball  $\overline{B_r(0)}$  of radius
2Here, the norms are taken on  $H^{s}(\mathbb{R}^{d})$  since the respective functions can be extended to  $\mathbb{R}^d$  by zero outside of their domain of definition, thanks to the properties of the partition of unity.
$1 < r < r_{1}$ . Hence, we can find  $\varphi \in C^{\infty}(\mathbb{R}^{d})$  with  $\varphi(B_{1}(0)) = \{1\}$  and  $\varphi((\overline{B_{r}(0)})^{c}) = \{0\}$ . Since  $\varphi$  is smooth with compact support, we have  $\varphi \cdot \bar{f} \in H^{s}(\mathbb{R}^{d})$ . Define
$$
f _ {\mathbb {S} ^ {d}}: \mathbb {S} ^ {d} \to \mathbb {R}, \boldsymbol {x} \mapsto \left\{ \begin{array}{l l} (\varphi \cdot \bar {f}) (\phi (\boldsymbol {x})) & , \boldsymbol {x} \in U _ {1} \\ 0 & , \boldsymbol {x} \not \in U _ {1}  . \end{array} \right.
$$
By construction, we have  $f_{\mathbb{S}^d}(\pmb{x}) = 0$  for all  $\pmb{x} \in U_2$ . Hence, the equivalent Sobolev norm from Step 4.1 is
$$
\begin{array}{l} \left\| f _ {\mathbb {S} ^ {d}} \right\| _ {H ^ {s} (\mathbb {S} ^ {d})} = \left(\left\| \left(h _ {1} f _ {\mathbb {S} ^ {d}}\right) \circ \phi_ {1} ^ {- 1} \right\| _ {H ^ {s} (\mathbb {R} ^ {d})} ^ {2} + \left\| \left(h _ {2} f _ {\mathbb {S} ^ {d}}\right) \circ \phi_ {2} ^ {- 1} \right\| _ {H ^ {s} (\mathbb {R} ^ {d})} ^ {2}\right) ^ {1 / 2} \\ = \left\| \left(h _ {1} \circ \phi_ {1} ^ {- 1}\right) \cdot \varphi \cdot \bar {f} \right\| _ {H ^ {s} (\mathbb {R} ^ {d})} <   \infty , \\ \end{array}
$$
which shows  $f_{\mathbb{S}^d}\in H^s (\mathbb{S}^d)$  . But then,  $f\circ \phi = f_{\mathbb{S}^d}|_T\in H^s (\mathbb{S}^d)|_T$
In total, we obtain  $H^{s}(\mathbb{S}^{d})|_{T} = H^{s}(B_{1}(0))\circ \phi$ , which shows (IV).
F Spectral lower bound
F.1 General lower bounds
A common first step to analyze the expected excess risk caused by label noise is to perform a bias-variance decomposition and integrate over  $y$  first (see e.g. Liang and Rakhlin, 2020, Hastie et al., 2022, Holzmuller, 2021), which is also used in the following lemma.
Lemma F.1. Consider an estimator of the form  $f_{\pmb{X},\pmb{y}}(\pmb{x}) = (\pmb{v}_{\pmb{X},\pmb{x}})^{\top}\pmb{y}$ . If  $\operatorname{Var}_P(y|\pmb{x}) \geq \sigma^2$  for  $P_X$ -almost all  $\pmb{x}$ , then the expected excess risk satisfies
$$
\mathbb {E} _ {D} R _ {P} (f _ {\boldsymbol {X}, \boldsymbol {y}}) - R _ {P} ^ {*} \geq \sigma^ {2} \mathbb {E} _ {\boldsymbol {X}, \boldsymbol {x}} \operatorname {t r} (\boldsymbol {v} _ {\boldsymbol {X}, \boldsymbol {x}} (\boldsymbol {v} _ {\boldsymbol {X}, \boldsymbol {x}}) ^ {\top}).
$$
Proof. A standard bias-variance decomposition lets us lower-bound the expected excess risk by the estimator variance due to the label noise, which can then be further simplified:
$$
\begin{array}{l} \mathbb {E} _ {D} R _ {P} \left(f _ {\boldsymbol {X}, \boldsymbol {y}}\right) - R _ {P} ^ {*} \geq \mathbb {E} _ {\boldsymbol {X}, \boldsymbol {x}} \left(\mathbb {E} _ {\boldsymbol {y} | \boldsymbol {X}} \left[ f _ {\boldsymbol {X}, \boldsymbol {y}} (\boldsymbol {x}) ^ {2} \right] - \left(\mathbb {E} _ {\boldsymbol {y} | \boldsymbol {X}} \left[ f _ {\boldsymbol {X}, \boldsymbol {y}} (\boldsymbol {x}) \right]\right) ^ {2}\right). \\ = \mathbb {E} _ {\boldsymbol {X}, \boldsymbol {x}} \mathbb {E} _ {\boldsymbol {y} | \boldsymbol {X}} \left(f _ {\boldsymbol {X}, \boldsymbol {y}} (\boldsymbol {x}) - \mathbb {E} _ {\boldsymbol {y} | \boldsymbol {X}} f _ {\boldsymbol {X}, \boldsymbol {y}} (\boldsymbol {x})\right) ^ {2} \\ = \mathbb {E} _ {\boldsymbol {X}, \boldsymbol {x}} \mathbb {E} _ {\boldsymbol {y} | \boldsymbol {X}} (\boldsymbol {v} _ {\boldsymbol {X}, \boldsymbol {x}}) ^ {\top} (\boldsymbol {y} - \mathbb {E} _ {\boldsymbol {y} | \boldsymbol {X}} \boldsymbol {y}) (\boldsymbol {y} - \mathbb {E} _ {\boldsymbol {y} | \boldsymbol {X}} \boldsymbol {y}) ^ {\top} \boldsymbol {v} _ {\boldsymbol {X}, \boldsymbol {x}} \\ = \mathbb {E} _ {\boldsymbol {X}, \boldsymbol {x}} \left(\boldsymbol {v} _ {\boldsymbol {X}, \boldsymbol {x}}\right) ^ {\top} \left[ \mathbb {E} _ {\boldsymbol {y} | \boldsymbol {X}} \left(\boldsymbol {y} - \mathbb {E} _ {\boldsymbol {y} | \boldsymbol {X}} \boldsymbol {y}\right) \left(\boldsymbol {y} - \mathbb {E} _ {\boldsymbol {y} | \boldsymbol {X}} \boldsymbol {y}\right) ^ {\top} \right] \boldsymbol {v} _ {\boldsymbol {X}, \boldsymbol {x}} \\ = \mathbb {E} _ {\boldsymbol {X}, \boldsymbol {x}} (\boldsymbol {v} _ {\boldsymbol {X}, \boldsymbol {x}}) ^ {\top} \operatorname {C o v} (\boldsymbol {y} | \boldsymbol {X}) \boldsymbol {v} _ {\boldsymbol {X}, \boldsymbol {x}}. \\ \end{array}
$$
Here, the conditional covariance matrix can be lower bounded in terms of the Loewner order (which is defined as  $A \succeq B \Leftrightarrow B - A$  positive semi-definite):
$$
\operatorname {C o v} (\boldsymbol {y} | \boldsymbol {X}) = \left( \begin{array}{c c c} \operatorname {V a r} (y _ {1} | \boldsymbol {x} _ {1}) & & \\ & \ddots & \\ & & \operatorname {V a r} (y _ {n} | \boldsymbol {x} _ {n}) \end{array} \right) \succeq \sigma^ {2} \boldsymbol {I} _ {n}
$$
since the labels  $y_{i}$  are conditionally independent given  $\mathbf{X}$ . We therefore obtain
$$
\begin{array}{l} \mathbb {E} _ {D} R _ {P} \left(f _ {\boldsymbol {X}, \boldsymbol {y}}\right) - R _ {P} ^ {*} \geq \mathbb {E} _ {\boldsymbol {X}, \boldsymbol {x}} \left(\boldsymbol {v} _ {\boldsymbol {X}, \boldsymbol {x}}\right) ^ {\top} \operatorname {C o v} (\boldsymbol {y} | \boldsymbol {X}) \boldsymbol {v} _ {\boldsymbol {X}, \boldsymbol {x}} \\ \geq \sigma^ {2} \mathbb {E} _ {\boldsymbol {X}, \boldsymbol {x}} \operatorname {t r} \left(\left(\boldsymbol {v} _ {\boldsymbol {X}, \boldsymbol {x}}\right) ^ {\top} \boldsymbol {v} _ {\boldsymbol {X}, \boldsymbol {x}}\right) \\ = \sigma^ {2} \mathbb {E} _ {\boldsymbol {X}, \boldsymbol {x}} \operatorname {t r} \left(\boldsymbol {v} _ {\boldsymbol {X}, \boldsymbol {x}} \left(\boldsymbol {v} _ {\boldsymbol {X}, \boldsymbol {x}}\right) ^ {\top}\right). \\ \end{array}
$$