SlowGuess/NeurIPS_2023_OCR huggingface dataset:
More precisely, the second condition means that for any two smooth functions  $\phi : \mathcal{M}_0 \to \mathbb{R}$  and  $\tilde{\phi} : \mathcal{M}_0 / G \to \mathbb{R}$  such that  $\phi(x) = \tilde{\phi}([x])$ , for any  $x \in \mathcal{M}_0$  and  $[x] \in \mathcal{M}_0 / G$ , the function  $\tilde{\phi}$  satisfies the Neumann boundary condition on  $\partial (\widetilde{\mathcal{M}})$ . Given this claim, by the local Weyl's law (Proposition A.1), the proof is complete.
We only need to specify the manifold  $\widetilde{\mathcal{M}}$  and prove that each projected invariant function on it satisfies the Neumann boundary condition. Indeed, the construction of the manifold  $\widetilde{\mathcal{M}}$  follows from the finite decomposition of the quotient space  $\mathcal{M} / G = \bigsqcup_{[H] \preceq G} \mathcal{M}_{[H]} / G$ . Moreover, we can assume that the boundary of  $\widetilde{\mathcal{M}}$  is piecewise smooth. Let  $[x] \in \partial (\widetilde{\mathcal{M}})$  be a boundary point (in the interior of a smooth piece of the boundary). We claim that
$$
\phi \text {i s} G - \text {i n v a r i a n t} \mathcal {M} \Rightarrow \langle \nabla_ {\tilde {g}} \tilde {\phi} ([ x ]), \hat {n} _ {[ x ]} \rangle_ {\tilde {g}} = 0, \tag {75}
$$
for any smooth  $\phi : \mathcal{M} \to \mathbb{R}$ , where  $\phi(x) = \tilde{\phi}([x])$ , and this completes the proof. Note that  $\hat{n}_{[x]} \in T_{[x]}(\widetilde{\mathcal{M}})$  is the unit outward normal vector of the manifold  $\widetilde{\mathcal{M}}$  at  $[x]$ . To prove the claim, we write  $T_{[x]}(\widetilde{\mathcal{M}}) = \operatorname{span}(\hat{n}_{[x]}) \oplus H_{[x]}$  for an orthogonal vector space  $H_{[x]}$ . But  $H_{[x]} \simeq T_{[x]}\partial(\widetilde{\mathcal{M}})$ . Also, in a neighborhood  $\mathcal{N}$  around  $[x]$  in  $\partial(\widetilde{\mathcal{M}})$ , for each  $[y] \in \mathcal{N}$ , we have the smooth identity  $(\rho_{[y]}^{-1} \circ \tau_{[x]} \circ \rho_{[y]})(y) = y$  for some  $\rho_{[y]}, \tau_{[x]} \in G$ , such that  $(\rho_{[y]}^{-1} \circ \tau_{[x]} \circ \rho_{[y]})$  does not belong to isotropy groups of  $\mathcal{M}_0 / G$  near  $[x]$ . Without loss of generality, we assume that  $\rho_{[x]} = \mathrm{id}_G$ .
Now consider a geodesic on  $\mathcal{M}$  starting from  $x\in \mathcal{M}$  with unit velocity such as  $\gamma (t)$  with  $\gamma (0) = x$  and  $d\pi_{[x]}(\gamma '(0)) = \hat{n}_{[x]}$ . Note that  $[\gamma (t)]\notin \partial (\widetilde{M})$  for small enough  $t\in (0,\epsilon)$ , and thus it belongs to  $\mathcal{M}_0 / G$ . But it is simultaneously "on the other side" of the particular fundamental domain of  $\mathcal{M}_0 / G$  around  $[x]$ , meaning that  $[\tau_{[x]}(\gamma (t))]$  is necessarily a curve starting from  $[x]$  towards the inside of the fundamental domain. In particular, since  $\tau_{[x]}$  is an isometry (and thus a local isometry), we necessarily have  $(\tau_{[x]}\circ \gamma)'(0) = -\hat{n}_{[x]}$  (note that in this step we clearly use the explanations in the previous paragraph). Now by considering the function  $\phi \circ \gamma = \phi \circ \tau_{[x]}\circ \gamma$  on the interval  $t\in (0,\epsilon)$ , we get
$$
\langle \nabla_ {\tilde {g}} \tilde {\phi} ([ x ]), \hat {n} _ {[ x ]} \rangle_ {\tilde {g}} = \langle \nabla_ {\tilde {g}} \tilde {\phi} ([ x ]), - \hat {n} _ {[ x ]} \rangle_ {\tilde {g}} \Longrightarrow \langle \nabla_ {\tilde {g}} \tilde {\phi} ([ x ]), \hat {n} _ {[ x ]} \rangle_ {\tilde {g}} = 0, \tag {76}
$$
In this section, we use Theorem 4.4 to prove Theorem 4.1. Let us first state a standard bound in the literature holding for any RKHS.
Proposition C.1 ([4], Proposition 7.3). Consider the KRR problem in an RKHS setting, and let  $f_{\mathrm{proj}}^{\star}$  denote the orthogonal projection of  $f^{\star}$  on the Hilbert space  $\mathcal{H}$ . Assume that  $K(x,x)\leq R^{2}$  for any  $x\in \mathcal{M}$ ,  $\eta \leq R^2$ , and  $n\geq \frac{5R^2}{\eta} (1 + \log (\frac{R^2}{\eta}))$ . Then,
$$
\begin{array}{l} \mathbb {E} \left[ \mathcal {R} (\hat {f}) - \mathcal {R} \left(f _ {\operatorname {p r o j}} ^ {\star}\right) \right] \leq 1 6 \frac {\sigma^ {2}}{n} \operatorname {t r} \left[ \left(\Sigma + \eta I\right) ^ {- 1} \Sigma \right] (77) \\ + 1 6 \inf  _ {f \in \mathcal {H}} \left\{\| f - f _ {\operatorname {p r o j}} ^ {\star} \| _ {L ^ {2} (\mathcal {M})} ^ {2} + \eta \| f \| _ {\mathcal {H}} ^ {2} \right\} + \frac {2 4}{n ^ {2}} \| f ^ {\star} \| _ {L ^ {\infty} (\mathcal {M})} ^ {2}, (78) \\ \end{array}
$$
where the expectation is over the randomness of the dataset  $S$ , and  $\Sigma = \mathbb{E}_{x\sim \mu}[\Phi_x\otimes_{\mathcal{H}}\Phi_x]$  is the covariance operator with the feature map  $\Phi_x = \sum_{\ell = 0}^{\infty}\mu_\ell \phi_\ell (x)\phi_\ell$  for any  $x\in \mathcal{M}$ .
Note that  $f_{\mathrm{proj}}^{\star} = f^{\star}$  if the closure of  $\mathcal{H}$  with respect to the  $L^2(\mathcal{M})$ -norm is  $L_{\mathrm{inv}}^2(\mathcal{M}, G)$ . In the Laplace-Beltrami basis,  $\Sigma$  is diagonal with the diagonal elements  $(\Sigma)_{\ell, \ell} = \mu_\ell$  for each  $\ell$ . Note that the first and second terms in the above upper bound are known as the variance and the bias terms, respectively. Also, while the bound holds in expectation for a random dataset  $S$ , assuming  $\epsilon_i$ 's are sub-Gaussian, one can extend the result to a high-probability bound using standard concentration inequalities. However, for the brevity/clarity of the paper, we restrict our attention to the expectation of the population risk.
We need to have an explicit upper bound for  $R$  to use the above proposition. Although the problem is essentially homogenous with respect to  $R$ , for the sake of completeness, we explicitly compute a uniform upper bound on the diagonal values of the kernel in terms of the problem's parameters. The goal is to first check that the two conditions  $R < \infty$  and  $n \geq \frac{5R^2}{\eta} (1 + \log (\frac{R^2}{\eta}))$  are satisfied. The latter condition is indeed satisfied when  $\eta \geq \frac{5R^2\log(n)}{n}$ . Note that if  $\mu_{\lambda,\ell} \neq 0$  for any  $\lambda, \ell$  with  $\ell = 1,2,\ldots, \dim(V_{\lambda,G})$ , then any  $G$ -invariant function  $f^{\star} \in \mathcal{F} \subseteq L_{\mathrm{inv}}^{2}(\mathcal{M},G)$  is in the closure of  $\mathcal{H}$ . Indeed, in that case the closure of  $\mathcal{H}$  with respect to the  $L^2(\mathcal{M})$ -norm includes  $L_{\mathrm{inv}}^{2}(\mathcal{M},G)$ .
C.1 Bounding  $K(x, x)$
We start with the definition of  $R$ ; for any  $x \in \mathcal{M}$ , we have
$$
K (x, x) = \left\langle \Phi_ {x}, \Phi_ {x} \right\rangle_ {\mathcal {H}} = \sum_ {\lambda \in \operatorname {S p e c} (\mathcal {M})} \sum_ {\ell = 1} ^ {\dim (V _ {\lambda , G})} \mu_ {\lambda , \ell} | \phi_ {\lambda , \ell} (x) | ^ {2}. \tag {79}
$$
By the local version of Theorem 4.4, we know that  $N_{x}(\lambda; G) = \sum_{\lambda' \leq \lambda} \sum_{\ell=1}^{\dim(V_{\lambda', G})} |\phi_{\lambda', \ell}(x)|^{2} \leq \frac{\omega_{d}}{(2\pi)^{d}} \operatorname{vol}(\mathcal{M}/G) \lambda^{d/2} + C_{\mathcal{M}/G} \lambda^{\frac{d-1}{2}}$ , for an absolute constant  $C_{\mathcal{M}/G}$ , where  $d$  denotes the effective dimension of the quotient space. Therefore, if  $\mu_{\lambda, \ell} \leq u(\lambda)$  for a differentiable bounded function  $u(\lambda)$ , for any  $\lambda, \ell$ , then
$$
\begin{array}{l} K (x, x) \leq \int_ {0 ^ {-}} ^ {\infty} u (\lambda) d N _ {x} (\lambda ; G) (80) \\ \stackrel {(a)} {=} \lim  _ {\lambda \rightarrow \infty} u (\lambda) N _ {x} (\lambda ; G) - u (0 ^ {-}) N _ {x} (0 ^ {-}; G) - \int_ {0 ^ {-}} ^ {\infty} N _ {x} (\lambda ; G) u ^ {\prime} (\lambda) d \lambda (81) \\ \stackrel {(b)} {\leq} \frac {- \omega_ {d}}{(2 \pi) ^ {d}} \operatorname {v o l} (\mathcal {M} / G) \int_ {0 ^ {-}} ^ {\infty} \lambda^ {d / 2} u ^ {\prime} (\lambda) d \lambda + C _ {\mathcal {M} / G} \int_ {0 ^ {-}} ^ {\infty} \lambda^ {(d - 1) / 2} u ^ {\prime} (\lambda) d \lambda (82) \\ \stackrel {(c)} {=} \frac {\omega_ {d}}{(2 \pi) ^ {d}} \operatorname {v o l} (\mathcal {M} / G) \frac {d}{2} \int_ {0 ^ {-}} ^ {\infty} \lambda^ {d / 2 - 1} u (\lambda) d \lambda + C _ {\mathcal {M} / G} \int_ {0 ^ {-}} ^ {\infty} \lambda^ {(d - 1) / 2 - 1} u (\lambda) d \lambda (83) \\ = \frac {d}{2} \frac {\omega_ {d}}{(2 \pi) ^ {d}} \operatorname {v o l} (\mathcal {M} / G) \left(\{\mathcal {M} u \} (d / 2)\right) + C _ {\mathcal {M} / G} \left(\left\{\mathcal {M} u \right\} ((d - 1) / 2)\right), (84) \\ \end{array}
$$
where (a) and (c) follow by integration by parts, and (b) follows from Theorem 4.4. The Mellin transform is defined as  $\{\mathcal{M}u\}(s) \coloneqq \int_0^\infty t^{s - 1}u(t)dt$ . Also, integration with respect to  $dN_{x}(\lambda;G)$  must be understood as a Riemann-Stieltjes integral.
C.2 Bounding the Bias Term
We have already observed that the function achieving the infimum is
$$
\hat {f} _ {\text {e f f}} = \sum_ {\lambda \in \operatorname {S p e c} (\mathcal {M})} \sum_ {\ell = 1} ^ {\dim (V _ {\lambda , G})} \frac {\mu_ {\lambda , \ell}}{\mu_ {\lambda , \ell} + \eta} \left\langle f ^ {\star}, \phi_ {\lambda , \ell} \right\rangle_ {L ^ {2} (\mathcal {M})} \phi_ {\lambda , \ell}. \tag {85}
$$
Note that clearly  $\hat{f}_{\mathrm{eff}} \in \mathcal{H}$  for  $\eta > 0$ , as we can explicitly compute  $\| \hat{f}_{\mathrm{eff}} \|_{\mathcal{H}} < \infty$ . Also,
$$
f _ {\operatorname {p r o j}} ^ {\star} = \sum_ {\lambda \in \operatorname {S p e c} (\mathcal {M})} \sum_ {\ell = 1} ^ {\dim (V _ {\lambda , G})} \mathbb {1} \left\{\mu_ {\lambda , \ell} \neq 0 \right\} \langle f ^ {\star}, \phi_ {\lambda , \ell} \rangle_ {L ^ {2} (\mathcal {M})} \phi_ {\lambda , \ell}. \tag {86}
$$
$$
\begin{array}{l} 1 6 \inf  _ {f \in \mathcal {H}} \left\{\| f - f _ {\operatorname {p r o j}} ^ {\star} \| _ {L ^ {2} (\mathcal {M})} ^ {2} + \eta \| f \| _ {\mathcal {H}} ^ {2} \right\} = 1 6 \| \hat {f} _ {\text {e f f}} - f _ {\operatorname {p r o j}} ^ {\star} \| _ {L ^ {2} (\mathcal {M})} ^ {2} + 1 6 \eta \| \hat {f} _ {\text {e f f}} \| _ {\mathcal {H}} ^ {2} (87) \\ = 1 6 \sum_ {\lambda \in \operatorname {S p e c} (\mathcal {M})} \sum_ {\ell = 1} ^ {\dim (V _ {\lambda , G})} \left(\frac {\mu_ {\lambda , \ell}}{\mu_ {\lambda , \ell} + \eta} - 1\right) ^ {2} \langle f _ {\operatorname {p r o j}} ^ {\star}, \phi_ {\lambda , \ell} \rangle_ {L ^ {2} (\mathcal {M})} ^ {2} (88) \\ \end{array}
$$
$$
\begin{array}{l} + 1 6 \eta \sum_ {\lambda \in \operatorname {S p e c} (\mathcal {M})} \sum_ {\ell = 1} ^ {\dim (V _ {\lambda , G})} \frac {1}{\mu_ {\lambda , \ell}} \left(\frac {\mu_ {\lambda , \ell}}{\mu_ {\lambda , \ell} + \eta} \langle f _ {\text {p r o j}} ^ {\star}, \phi_ {\lambda , \ell} \rangle_ {L ^ {2} (\mathcal {M})}\right) ^ {2} (89) \\ = 1 6 \sum_ {\lambda \in \operatorname {S p e c} (\mathcal {M})} \sum_ {\ell = 1} ^ {\dim (V _ {\lambda , G})} \left(\frac {\eta^ {2} + \eta \mu_ {\lambda , \ell}}{(\mu_ {\lambda , \ell} + \eta) ^ {2}}\right) \langle f _ {\text {p r o j}} ^ {\star}, \phi_ {\lambda , \ell} \rangle_ {L ^ {2} (\mathcal {M})} ^ {2} (90) \\ = 1 6 \eta \sum_ {\lambda \in \operatorname {S p e c} (\mathcal {M})} \sum_ {\ell = 1} ^ {\dim (V _ {\lambda , G})} \left(\frac {1}{\mu_ {\lambda , \ell} + \eta}\right) \langle f _ {\operatorname {p r o j}} ^ {\star}, \phi_ {\lambda , \ell} \rangle_ {L ^ {2} (\mathcal {M})} ^ {2}. (91) \\ \end{array}
$$
C.3 Bounding the Variance Term
We need to compute the trace of the operator  $(\Sigma + \eta I)^{-1}\Sigma$ . But this operator is diagonal in the Laplace-Beltrami basis, and thus we get
$$
1 6 \frac {\sigma^ {2}}{n} \operatorname {t r} [ (\Sigma + \eta I) ^ {- 1} \Sigma ] = 1 6 \frac {\sigma^ {2}}{n} \sum_ {\lambda \in \operatorname {S p e c} (\mathcal {M})} \sum_ {\ell = 1} ^ {\dim (V _ {\lambda , G})} \frac {\mu_ {\lambda , \ell}}{\mu_ {\lambda , \ell} + \eta}. \tag {92}
$$
C.4 Bounding the Population Risk
Now we combine the previous steps to get
$$
\begin{array}{l} \mathbb {E} \left[ \mathcal {R} (\hat {f}) - \mathcal {R} \left(f _ {\operatorname {p r o j}} ^ {\star}\right) \right] \leq 1 6 \frac {\sigma^ {2}}{n} \sum_ {\lambda \in \operatorname {S p e c} (\mathcal {M})} \sum_ {\ell = 1} ^ {\dim (V _ {\lambda , G})} \frac {\mu_ {\lambda , \ell}}{\mu_ {\lambda , \ell} + \eta} (93) \\ + 1 6 \eta \sum_ {\lambda \in \operatorname {S p e c} (\mathcal {M})} \sum_ {\ell = 1} ^ {\dim (V _ {\lambda , G})} \left(\frac {1}{\mu_ {\lambda , \ell} + \eta}\right) \langle f _ {\operatorname {p r o j}} ^ {\star}, \phi_ {\lambda , \ell} \rangle_ {L ^ {2} (\mathcal {M})} ^ {2} (94) \\ + \frac {2 4}{n ^ {2}} \| f ^ {\star} \| _ {L ^ {\infty} (\mathcal {M})} ^ {2}, (95) \\ \end{array}
$$
which holds when  $R < \infty$  and  $\eta \geq \frac{5R^2\log(n)}{n}$ .
We are now ready to bound the convergence rate of the population risk of KRR for invariant Sobolev space  $\mathcal{H}_{\mathrm{inv}}^s (\mathcal{M})$  (See Section A.10 for the definition). In this case,  $\mu_{\lambda ,\ell} = u(\lambda) = \min (1,\lambda^{-s})$  for each  $\lambda ,\ell$ . Therefore,
$$
\left\{\mathcal {M} u \right\} (d / 2) = \int_ {0} ^ {\infty} \min  \left(1, \lambda^ {- s}\right) t ^ {d / 2 - 1} d t \leq 1 + \frac {1}{s - d / 2}. \tag {96}
$$
Similarly,  $\{\mathcal{M}u\}((d - 1)/2) \leq 1 + \frac{1}{s - (d - 1)/2}$ . Thus, using the analysis in Section C.1, we get
$$
\begin{array}{l} K _ {\mathcal {H} ^ {s} (\mathcal {M})} (x, x) \leq R ^ {2} := \frac {\omega_ {d}}{(2 \pi) ^ {d}} \operatorname {v o l} (\mathcal {M} / G) \left(d / 2 + \frac {d / 2}{s - d / 2}\right) (97) \\ + C _ {\mathcal {M} / G} \left(1 + \frac {1}{s - (d - 1) / 2}\right). (98) \\ \end{array}
$$
In particular,  $R < \infty$  if  $s > d / 2$ . We now compute the bias and the variance terms as follows. Let us start with the variance term:
$$
\begin{array}{l} 1 6 \frac {\sigma^ {2}}{n} \sum_ {\lambda \in \operatorname {S p e c} (\mathcal {M})} \sum_ {\ell = 1} ^ {\dim (V _ {\lambda , G})} \frac {\mu_ {\lambda , \ell}}{\mu_ {\lambda , \ell} + \eta} = 1 6 \frac {\sigma^ {2}}{n} \int_ {0 ^ {-}} ^ {\infty} \frac {\min  \left(1 , \lambda^ {- s}\right)}{\min  \left(1 , \lambda^ {- s}\right) + \eta} d N (\lambda ; G) (99) \\ \leq 1 6 \frac {\sigma^ {2}}{n} N (1; G) + 1 6 \frac {\sigma^ {2}}{n} \int_ {1} ^ {\infty} \frac {\lambda^ {- s}}{\lambda^ {- s} + \eta} d N (\lambda ; G) (100) \\ = 1 6 \frac {\sigma^ {2}}{n} N (1; G) + 1 6 \frac {\sigma^ {2}}{n} \int_ {1} ^ {\infty} \frac {1}{1 + \eta \lambda^ {s}} d N (\lambda ; G) (101) \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(a)} {=} 1 6 \frac {\sigma^ {2}}{n} N (1; G) + 1 6 \frac {\sigma^ {2}}{n} \lim  _ {\lambda \rightarrow \infty} \left(\frac {N (\lambda ; G)}{1 + \eta \lambda^ {s}}\right) - 1 6 \frac {\sigma^ {2}}{n} \frac {N (1 ; G)}{1 + \eta} (102) \\ + 1 6 \frac {\sigma^ {2}}{n} \int_ {1} ^ {\infty} N (\lambda ; G) \frac {s \eta \lambda^ {s - 1}}{\left(1 + \eta \lambda^ {s}\right) ^ {2}} d \lambda (103) \\ \stackrel {(b)} {=} 1 6 \frac {\sigma^ {2}}{n} \frac {\eta}{1 + \eta} N (1; G) + 1 6 \frac {\sigma^ {2}}{n} \int_ {1} ^ {\infty} N (\lambda ; G) \frac {s \eta \lambda^ {s - 1}}{(1 + \eta \lambda^ {s}) ^ {2}} d \lambda (104) \\ \stackrel {(c)} {=} 1 6 \frac {\sigma^ {2}}{n} \frac {\eta}{1 + \eta} N (1; G) + 1 6 \frac {\sigma^ {2}}{n} \frac {\omega_ {d}}{(2 \pi) ^ {d}} \operatorname {v o l} (\mathcal {M} / G) s \eta \int_ {1} ^ {\infty} \lambda^ {d / 2} \frac {\lambda^ {s - 1}}{(1 + \eta \lambda^ {s}) ^ {2}} d \lambda (105) \\ + 1 6 \frac {\sigma^ {2}}{n} C _ {\mathcal {M} / G} s \eta \int_ {1} ^ {\infty} \lambda^ {(d - 1) / 2} \frac {\lambda^ {s - 1}}{(1 + \eta \lambda^ {s}) ^ {2}} d \lambda (106) \\ \leq 1 6 \frac {\sigma^ {2}}{n} \eta N (1; G) + 1 6 \frac {\sigma^ {2}}{n} \frac {\omega_ {d}}{(2 \pi) ^ {d}} \operatorname {v o l} (\mathcal {M} / G) s \eta \int_ {1} ^ {\infty} \frac {\lambda^ {d / 2 + s - 1}}{1 + \eta^ {2} \lambda^ {2 s}} d \lambda (107) \\ + 1 6 \frac {\sigma^ {2}}{n} C _ {\mathcal {M} / G} s \eta \int_ {1} ^ {\infty} \frac {\lambda^ {(d - 1) / 2 + s - 1}}{1 + \eta^ {2} \lambda^ {2 s}} d \lambda , (108) \\ \end{array}
$$
where (a) follows by integration by parts, (b) follows by  $\lim_{\lambda \to \infty}\left(\frac{N(\lambda;G)}{1 + \eta\lambda^s}\right) = 0$  since  $s > d / 2$ , and (c) follows from Theorem 4.4. Note that integration with respect to  $dN(\lambda ;G)$  must be understood as a Riemann-Stieltjes integral. By a change of variable in the integrals as  $t = \lambda \eta^{1 / s}$ , we have
$$
\int_ {1} ^ {\infty} \frac {\lambda^ {d / 2 + s - 1}}{1 + \eta^ {2} \lambda^ {2 s}} d \lambda = \eta^ {\frac {- 1}{s} (s + d / 2 - 1)} \eta^ {- 1 / s} \int_ {1} ^ {\infty} \frac {t ^ {d / 2 + s - 1}}{1 + t ^ {2 s}} d t \leq \frac {\eta^ {\frac {- 1}{s} (s + d / 2)}}{2 s - d}. \tag {109}
$$
We can similarly evaluate the other integral and conclude
$$
\begin{array}{l} 1 6 \frac {\sigma^ {2}}{n} \sum_ {\lambda \in \operatorname {S p e c} (\mathcal {M})} \sum_ {\ell = 1} ^ {\dim (V _ {\lambda , G})} \frac {\mu_ {\lambda , \ell}}{\mu_ {\lambda , \ell} + \eta} \leq 1 6 \frac {\sigma^ {2}}{n} \eta N (1; G) (110) \\ + 1 6 \frac {s \sigma^ {2}}{(2 s - d) n} \frac {\omega_ {d}}{(2 \pi) ^ {d}} \operatorname {v o l} (\mathcal {M} / G) \eta^ {1 - \frac {1}{s} (s + d / 2)} (111) \\ + 1 6 \frac {s \sigma^ {2}}{(2 s - d + 1) n} C _ {\mathcal {M} / G} \eta^ {1 - \frac {1}{s} (s + (d - 1) / 2)}. (112) \\ \end{array}
$$
One can also use the bound  $N(1;G) \leq \frac{\omega_d}{(2\pi)^d} \operatorname{vol}(\mathcal{M}/G) + C_{\mathcal{M}/G}$  to simplify the upper bound.
Note that  $f_{\mathrm{proj}}^{\star} = f^{\star}$  since the closure of  $\mathcal{H}_{\mathrm{inv}}^{\mathrm{s}}(\mathcal{M})$  with respect to the  $L^2 (\mathcal{M})$ -norm is the whole space  $L_{\mathrm{inv}}^{2}(\mathcal{M},G)$ . Let us now analyze the bias term by noting that  $\mu_{\lambda ,\ell} + \eta \geq \mu_{\lambda ,\ell}^{\theta}\eta^{1 - \theta}$  for any  $\theta \in (0,1]$ , and thus
$$
\begin{array}{l} 1 6 \eta \sum_ {\lambda \in \operatorname {S p e c} (\mathcal {M})} \sum_ {\ell = 1} ^ {\dim (V _ {\lambda , G})} \left(\frac {1}{\mu_ {\lambda , \ell} + \eta}\right) \langle f ^ {\star}, \phi_ {\lambda , \ell} \rangle_ {L ^ {2} (\mathcal {M})} ^ {2} (113) \\ \leq 1 6 \eta^ {\theta} \sum_ {\lambda \in \operatorname {S p e c} (\mathcal {M})} \sum_ {\ell = 1} ^ {\dim (V _ {\lambda , G})} \mu_ {\lambda , \ell} ^ {- \theta} \langle f ^ {\star}, \phi_ {\lambda , \ell} \rangle_ {L ^ {2} (\mathcal {M})} ^ {2} (114) \\ = 1 6 \eta^ {\theta} \sum_ {\lambda \in \operatorname {S p e c} (\mathcal {M})} \sum_ {\ell = 1} ^ {\dim (V _ {\lambda , G})} \max  \left(1, \lambda^ {s \theta}\right) \langle f ^ {\star}, \phi_ {\lambda , \ell} \rangle_ {L ^ {2} (\mathcal {M})} ^ {2} (115) \\ = 1 6 \eta^ {\theta} \| f ^ {\star} \| _ {\mathcal {H} _ {\text {i n v}} ^ {s \theta} (\mathcal {M})} ^ {2}, (116) \\ \end{array}
$$
where  $\theta \in (0,1]$  is chosen so that  $f^{\star}\in \mathcal{H}_{\mathrm{inv}}^{s\theta}(\mathcal{M})$  . Therefore,
$$
\mathbb {E} \left[ \mathcal {R} (\hat {f}) - \mathcal {R} (f ^ {\star}) \right] \leq 1 6 \frac {\sigma^ {2}}{n} \eta \left(\frac {\omega_ {d}}{(2 \pi) ^ {d}} \operatorname {v o l} \left(\mathcal {M} / G\right) + C _ {\mathcal {M} / G}\right) \tag {117}
$$
$$
\begin{array}{l} + 1 6 \frac {s \sigma^ {2}}{(2 s - d) n} \frac {\omega_ {d}}{(2 \pi) ^ {d}} \operatorname {v o l} (\mathcal {M} / G) \eta^ {1 - \frac {1}{s} (s + d / 2)} (118) \\ + 1 6 \frac {s \sigma^ {2}}{(2 s - d + 1) n} C _ {\mathcal {M} / G} \eta^ {1 - \frac {1}{s} (s + (d - 1) / 2)} (119) \\ + 1 6 \eta^ {\theta} \| f ^ {\star} \| _ {\mathcal {H} _ {\text {i n v}} ^ {s \theta} (\mathcal {M})} ^ {2} + \frac {2 4}{n ^ {2}} \| f ^ {\star} \| _ {L ^ {\infty} (\mathcal {M})} ^ {2}, (120) \\ \end{array}
$$
The result is only true when  $R < \infty$  and  $\eta \geq \frac{5R^2\log(n)}{n}$ , where  $R$  is defined in Equation (98).
We can now optimize the above bound for  $\eta$ . First consider the function  $p(t) = c_a t^{-a} + c_b t^b$  on  $\mathbb{R}_{\geq 0}$  for  $a, b, c_a, c_b > 0$ . Note that  $p(0) = p(\infty) = \infty$ , and to find its stationary points, we solve  $p'(t) = 0$  and get the only solution  $t = \left(\frac{ac_a}{bc_b}\right)^{1/(a+b)}$  which is thus the global minimum of the function. Thus by minimizing
$$
p (\eta) = \left(1 6 \frac {s \sigma^ {2}}{(2 s - d) n} \frac {\omega_ {d}}{(2 \pi) ^ {d}} \operatorname {v o l} (\mathcal {M} / G)\right) \eta^ {- \frac {d}{2 s}} + \left(1 6 \| f ^ {\star} \| _ {\mathcal {H} _ {\mathrm {i n v}} ^ {s \theta} (\mathcal {M})} ^ {2}\right) \eta^ {\theta}, \tag {121}
$$
$$
\eta = \left(\frac {d \sigma^ {2}}{2 \theta \| f ^ {\star} \| _ {\mathcal {H} _ {\mathrm {i n v}} ^ {s \theta} (\mathcal {M})} ^ {2} (2 s - d) n} \frac {\omega_ {d}}{(2 \pi) ^ {d}} \operatorname {v o l} (\mathcal {M} / G)\right) ^ {s / (\theta s + d / 2)}. \tag {122}
$$
Therefore, the population risk is bounded as follows:
$$
\begin{array}{l} \mathbb {E} \left[ \mathcal {R} (\hat {f}) - \mathcal {R} \left(f ^ {\star}\right) \right] (123) \\ \leq \underbrace {1 6 \left(\frac {\omega_ {d}}{(2 \pi) ^ {d}} \operatorname {v o l} (\mathcal {M} / G) + C _ {\mathcal {M} / G}\right) \frac {\sigma^ {2}}{n} \left(\frac {d \sigma^ {2}}{2 \theta \| f ^ {\star} \| _ {\mathcal {H} _ {\mathrm {i n v}} ^ {s \theta} (\mathcal {M})} ^ {2} (2 s - d) n} \frac {\omega_ {d}}{(2 \pi) ^ {d}} \operatorname {v o l} (\mathcal {M} / G)\right) ^ {s / (\theta s + d / 2)}} _ {\mathcal {O} \left(n ^ {- 1 - s / (\theta s + d / 2)}\right)} (124) \\ + \underbrace {1 6 \frac {s \sigma^ {2}}{(2 s - d + 1) n} C _ {\mathcal {M} / G} \left(\frac {d \sigma^ {2}}{2 \theta \| f ^ {\star} \| _ {\mathcal {H} _ {\mathrm {i n v}} ^ {s \theta} (\mathcal {M})} ^ {2} (2 s - d) n} \frac {\omega_ {d}}{(2 \pi) ^ {d}} \operatorname {v o l} (\mathcal {M} / G)\right) ^ {- (d - 1) / (2 \theta s + d)}} _ {\mathcal {O} \left(n ^ {- (\theta s + 1 / 2) / (\theta s + d / 2)}\right)} (125) \\ + \underbrace {3 2 \left(\frac {d \sigma^ {2}}{2 \theta (2 s - d) n} \frac {\omega_ {d}}{(2 \pi) ^ {d}} \operatorname {v o l} (\mathcal {M} / G)\right) ^ {\theta s / (\theta s + d / 2)} \| f ^ {\star} \| _ {\mathcal {H} _ {\mathrm {i n v}} ^ {s \theta} (\mathcal {M})} ^ {d / (\theta s + d / 2)}} _ {\mathcal {O} \left(n ^ {- \theta s / (\theta s + d / 2)}\right)} (126) \\ + \underbrace {\frac {2 4}{n ^ {2}} \| f ^ {\star} \| _ {L ^ {\infty} (\mathcal {M})} ^ {2}} _ {\mathcal {O} (1 / n ^ {2})}, (127) \\ \end{array}
$$
and this completes the proof since  $s = \frac{d}{2} (\kappa + 1)$  and the third term dominates the sum.
D Proofs of Theorem 4.6, Proposition 4.5, and Corollary 4.7
Corollary 4.7 follows from Theorem 4.6 on the dimension of the vector space  $\mathcal{H}_G$ , just according to standard bounds in the literature on the population risk of KRR for finite-rank kernels (see [65]). Therefore, we focus on the proof of Theorem 4.6 and Proposition 4.5.
Proof of Proposition 4.5. This is a classical result; one can use a variational method to prove it. See [12] for the proof for an arbitrary vector space  $V$ . Here, we prove it for  $V = \mathcal{H}_G$ . Consider an arbitrary  $f = \sum_{\ell=0}^{D-1} \langle f, \phi_\ell \rangle_{L^2(\mathcal{M})} \phi_\ell \in \mathcal{H}_G$ . Then, using Equation (27),
$$
\mathcal {E} (f, f) = \int_ {\mathcal {M}} | \nabla_ {g} f (x) | _ {g} ^ {2} d \operatorname {v o l} _ {g} (x) \tag {128}
$$
$$
\begin{array}{l} = \sum_ {\ell = 0} ^ {D - 1} \lambda_ {\ell} | \langle f, \phi_ {\ell} \rangle_ {L ^ {2} (\mathcal {M})} | ^ {2} (129) \\ \leq \lambda_ {D - 1} \sum_ {\ell = 0} ^ {D - 1} \left\langle f, \phi_ {\ell} \right\rangle_ {L ^ {2} (\mathcal {M})} ^ {2} (130) \\ = \lambda_ {D - 1} \| f \| _ {L ^ {2} (\mathcal {M})} ^ {2}, (131) \\ \end{array}
$$
and the bound is achieved by  $f = \phi_{D - 1}$ .
Now by Theorem 4.4, the proof of Theorem 4.6 is also complete.
E Proof of Theorem 4.3
In this section, we mostly use/follow standard results in the literature of minimax lower bounds which can be found in [65]. Note that the unit ball in the Sobolev space  $\mathcal{H}_{\mathrm{inv}}^s (\mathcal{M})$  is isomorphic to the following ellipsoid (see Appendix A.10):