SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\left\| T _ {\lambda} ^ {- \frac {1}{2}} \left[ \left(\tilde {g} _ {Z} - T _ {X} f _ {\lambda}\right) - (g - T f _ {\lambda}) \right] \right\| = o _ {\mathbb {P}} \left(\lambda^ {\tilde {s} / 2}\right) = o _ {\mathbb {P}} \left(n ^ {- \tilde {s} \theta / 2}\right). \tag {43}
$$
Before proving Lemma A.5, we have to introduce the following proposition bounding the RKHS norm of the regularized basis function, which is a part of Li et al. [2023a, Corollary 5.6].
Proposition A.6. Suppose  $\mathcal{H}$  has embedding index  $\alpha_0$ . Then for any  $\alpha > \alpha_0$
$$
\left\| T _ {\lambda} ^ {- 1 / 2} k (x, \cdot) \right\| _ {\mathcal {H}} \leq M _ {\alpha} \lambda^ {- \alpha / 2}, \quad \mu \text {- a . e .} x \in \mathcal {X}. \tag {44}
$$
Proof of Lemma A.5. To use Bernstein inequality in Lemma B.4, let us bound the  $m$ -th moment of  $\xi(x)$ :
$$
\begin{array}{l} \mathbb {E} \| \xi (x) \| _ {\mathcal {H}} ^ {m} = \mathbb {E} \left\| T _ {\lambda} ^ {- \frac {1}{2}} K _ {x} \left(f _ {\rho} ^ {*} (x) - f _ {\lambda} (x)\right) \right\| _ {\mathcal {H}} ^ {m} \\ \leq \mathbb {E} \left[ \left\| T _ {\lambda} ^ {- \frac {1}{2}} k (x, \cdot) \right\| _ {\mathcal {H}} ^ {m} \cdot \mathbb {E} \left(\left| f _ {\rho} ^ {*} (x) - f _ {\lambda} (x) \right| ^ {m} | x\right) \right]. \tag {45} \\ \end{array}
$$
The first term in (45) is bounded through (44). For the second term, since  $s > \alpha_0$ , using the embedding condition and Lemma A.3, we have
$$
\left\| f _ {\lambda} - f _ {\rho} ^ {*} \right\| _ {L ^ {\infty}} \leq M _ {\alpha} \left\| f _ {\lambda} - f _ {\rho} ^ {*} \right\| _ {[ \mathcal {H} ] ^ {\alpha}} \leq C M _ {\alpha} \lambda^ {\min  (s - \alpha , 2) / 2} \leq C M _ {\alpha} \lambda^ {(\tilde {s} - \alpha) / 2},
$$
where we notice that  $\min (s - \alpha ,2) = \min (s,2 + \alpha) - \alpha \geq \tilde{s} -\alpha$  for the last inequality. Moreover, Lemma A.3 also implies
$$
\mathbb {E} \left| f _ {\lambda} (x) - f _ {\rho} ^ {*} (x) \right| ^ {2} = \left\| f _ {\lambda} (x) - f _ {\rho} ^ {*} (x) \right\| _ {L ^ {2}} ^ {2} \leq C \lambda^ {\bar {s}} \ln \frac {1}{\lambda} \leq C \lambda^ {\bar {s}} \ln n.
$$
Plugging in these estimations in (45), we get
$$
\begin{array}{l} (4 5) \leq \left(M _ {\alpha} \lambda^ {- \alpha / 2}\right) ^ {m} \cdot \left\| f _ {\lambda} - f _ {\rho} ^ {*} \right\| _ {L ^ {\infty}} ^ {m - 2} \cdot \mathbb {E} \left| f _ {\lambda} (x) - f _ {\rho} ^ {*} (x) \right| ^ {2} \\ \leq \left(M _ {\alpha} \lambda^ {- \alpha / 2}\right) ^ {m} \cdot \left(C M _ {\alpha} \lambda^ {(\tilde {s} - \alpha) / 2}\right) ^ {m - 2} \cdot \left(C \lambda^ {\tilde {s}} \ln n\right) \\ \leq \frac {1}{2} m! \left(C M _ {\alpha} ^ {2} \lambda^ {\tilde {s} - \alpha} \ln n\right) \cdot \left(C M _ {\alpha} ^ {2} \lambda^ {- \alpha + \tilde {s} / 2}\right) ^ {m - 2}. \tag {46} \\ \end{array}
$$
The proof is then complete by Lemma B.4.
The case of  $s \leq \alpha_0$  is more difficult. We will use the truncation technique introduced in Zhang et al. [2023a]. The following lemma can be proven similarly to Lemma A.3.
Lemma A.7. Under Assumptions 1 and 3, for any  $0 \leq \gamma < s + 2$ , we have
$$
\left\| f _ {\lambda} \right\| _ {[ \mathcal {H} ] ^ {\gamma}} ^ {2} \asymp \left\{ \begin{array}{l l} \lambda^ {s - \gamma}, & s <   \gamma ; \\ \ln \frac {1}{\lambda}, & s = \gamma ; \\ 1, & s > \gamma . \end{array} \right. \tag {47}
$$
Proof. Simply notice that
$$
\| f _ {\lambda} \| _ {[ \mathcal {H} ] ^ {\gamma}} ^ {2} = \sum_ {i = 1} ^ {\infty} a _ {i} ^ {2} \frac {\lambda_ {i} ^ {2}}{(\lambda_ {i} + \lambda) ^ {2}} (\lambda_ {i} ^ {s} i ^ {- 1}) \lambda_ {i} ^ {- \gamma} \asymp \sum_ {i = 1} ^ {\infty} \left(\frac {\lambda_ {i} ^ {p}}{\lambda_ {i} + \lambda}\right) ^ {2} i ^ {- 1},
$$
where  $p = (s + 2 - \gamma) / 2$ . Then we can apply Proposition B.2.
Then, we are able to show the following concentration result about the truncated  $\xi_{i}$ 's, whose proof resembles that of Lemma A.5.
Lemma A.8. Suppose Assumptions 1-3 hold and  $s \leq \alpha_0$ . Let  $\lambda \asymp n^{-\theta}$  with  $\theta \in (0, \beta)$  and  $\delta \in (0, 1)$ . For any  $t > 0$ , denote  $\Omega_t = \{x \in \mathcal{X} : |f_\rho^*(x)| \leq t\}$  and  $\bar{\xi}(x) = \xi(x) \mathbf{1}_{\{x \in \Omega_t\}}$ . Then, for  $\alpha > \alpha_0 = \beta^{-1}$  being sufficiently close, it holds with probability at least  $1 - \delta$  that
$$
\left\| \frac {1}{n} \sum_ {i = 1} ^ {n} \bar {\xi} (x _ {i}) - \mathbb {E} \bar {\xi} (x) \right\| \leq C \ln \frac {2}{\delta} \cdot \left[ \frac {M _ {\alpha}}{n} \left(M _ {\alpha} \lambda^ {- \alpha} + t \lambda^ {- \frac {\alpha + s}{2}}\right) + M _ {\alpha} \sqrt {\frac {\lambda^ {- \alpha} \ln n}{n}} \right] \lambda^ {s / 2}. \tag {48}
$$
Consequently, if  $t \asymp n^l$  with  $l < 1 - \frac{\alpha + s}{2}\theta$ , we have
$$
\left\| \frac {1}{n} \sum_ {i = 1} ^ {n} \bar {\xi} \left(x _ {i}\right) - \mathbb {E} \bar {\xi} (x) \right\| = o _ {\mathbb {P}} \left(\lambda^ {s / 2}\right). \tag {49}
$$
Proof. We follow the same routine of the proof of Lemma A.5 and obtain (45) with  $\xi$  replaced with  $\bar{\xi}$ . The only difference is that we have to control
$$
\begin{array}{l} \left\| \mathbf {1} \{x \in \Omega_ {t} \} \left(f _ {\lambda} - f _ {\rho} ^ {*}\right) \right\| _ {L ^ {\infty}} \leq \left\| f _ {\lambda} \right\| _ {L ^ {\infty}} + \left\| \mathbf {1} \{x \in \Omega_ {t} \} f _ {\rho} ^ {*} \right\| _ {L ^ {\infty}} \\ \leq M _ {\alpha} \| f _ {\lambda} \| _ {[ \mathcal {H} ] ^ {\alpha}} + t \\ \leq C M _ {\alpha} \lambda^ {(s - \alpha) / 2} + t, \\ \end{array}
$$
where we apply Lemma A.7 at the second inequality. Then, (46) changes to
$$
\frac {1}{2} m! \left(C M _ {\alpha} ^ {2} \lambda^ {\bar {s} - \alpha} \ln n\right) \cdot \left(C M _ {\alpha} ^ {2} \lambda^ {- \alpha + \bar {s} / 2} + M _ {\alpha} \lambda^ {- \alpha / 2} t\right) ^ {m - 2}
$$
and the rest follows.
To bound the extra error terms caused by truncation, we have to use the following proposition about the  $L^q$  embedding of the RKHS [Zhang et al., 2023a, Theorem 5].
Proposition A.9. Under Assumption 2, for any  $0 < s \leq \alpha_0$  and  $\alpha > \alpha_0$ , we have embedding
$$
[ \mathcal {H} ] ^ {s} \hookrightarrow L ^ {q _ {s}} (\mathcal {X}, \mathrm {d} \mu), \quad q _ {s} = \frac {2 \alpha}{\alpha - s}. \tag {50}
$$
Lemma A.10. Suppose Assumptions 1-3 hold and  $s \leq \alpha_0$ . Let  $\lambda \asymp n^{-\theta}$  with  $\theta \in (0, \beta)$  and  $\delta \in (0, 1)$ . Then
$$
\left\| T _ {\lambda} ^ {- \frac {1}{2}} \left[ \left(\tilde {g} _ {Z} - T _ {X} f _ {\lambda}\right) - (g - T f _ {\lambda}) \right] \right\| = o _ {\mathbb {P}} \left(\lambda^ {s / 2}\right) = o _ {\mathbb {P}} \left(n ^ {- s \theta / 2}\right). \tag {51}
$$
Proof. We will choose  $t = n^l$  for some  $l$  that will be determined later and choose some  $\alpha > \alpha_0$  being sufficiently close. Using the same notations as in (49), we decompose
$$
\begin{array}{l} \left\| \frac {1}{n} \sum_ {i = 1} ^ {n} \xi \left(x _ {i}\right) - \mathbb {E} \xi (x) \right\| _ {\mathcal {H}} \leq \left\| \frac {1}{n} \sum_ {i = 1} ^ {n} \bar {\xi} \left(x _ {i}\right) - \mathbb {E} \bar {\xi} (x) \right\| _ {\mathcal {H}} + \left\| \frac {1}{n} \sum_ {i = 1} ^ {n} \xi \left(x _ {i}\right) \mathbf {1} _ {\{x _ {i} \notin \Omega_ {t} \}} \right\| _ {\mathcal {H}} \tag {52} \\ + \left\| \mathbb {E} \xi (x) \mathbf {1} _ {\{x \notin \Omega_ {t} \}} \right\| _ {\mathcal {H}}. \\ \end{array}
$$
The first term in (49) is already bounded by (49) if  $l < 1 - \frac{\alpha + s}{2}\theta$ . To bound the second term in (52), we notice that
$$
x _ {i} \in \Omega_ {t}, \forall i = 1, \dots , n \quad \text {i m p l i e s} \quad \frac {1}{n} \sum_ {i = 1} ^ {n} \xi (x _ {i}) \mathbf {1} _ {\{x _ {i} \notin \Omega_ {t} \}} = 0.
$$
Since Markov's inequality yields
$$
\mathbb {P} _ {x \sim \mu} \left\{x \notin \Omega_ {t} \right\} \leq t ^ {- q} \left\| f _ {\rho} ^ {*} \right\| _ {L ^ {q}} ^ {q}, \tag {53}
$$
where  $q = \frac{2\alpha}{\alpha - s}$ , we get
$$
\mathbb {P} \left\{x _ {i} \in \Omega_ {t}, \forall i \right\} = \left(\mathbb {P} _ {x \sim \mu} \left\{x \in \Omega_ {t} \right\}\right) ^ {n} = (1 - \mathbb {P} _ {x \sim \mu} \left\{x \notin \Omega_ {t} \right\}) ^ {n} \geq (1 - t ^ {- q} \left\| f _ {\rho} ^ {*} \right\| _ {L ^ {q}} ^ {q}) ^ {n}.
$$
So the second term vanishes with high probability as long as  $l > 1 / q$ .
For the third term in (52), using (44), we get
$$
\begin{array}{l} \left\| \mathbb {E} \xi (x) \mathbf {1} _ {\{x \notin \Omega_ {t} \}} \right\| _ {\mathcal {H}} \leq \mathbb {E} \left\| \xi (x) \mathbf {1} _ {\{x \notin \Omega_ {t} \}} \right\| _ {\mathcal {H}} \\ = \mathbb {E} \left[ \mathbf {1} _ {\{x \notin \Omega_ {t} \}} \left(f _ {\rho} ^ {*} (x) - f _ {\lambda} (x)\right) \| \left(T _ {\lambda}\right) ^ {- 1 / 2} k (x, \cdot) \| _ {\mathcal {H}} \right] \\ \leq M _ {\alpha} \lambda^ {- \alpha / 2} \mathbb {E} \left[ \mathbf {1} _ {\{x \notin \Omega_ {t} \}} \left(f _ {\rho} ^ {*} (x) - f _ {\lambda} (x)\right) \right] \\ \leq M _ {\alpha} \lambda^ {- \alpha / 2} \left[ \mathbb {E} \left(f _ {\rho} ^ {*} (x) - f _ {\lambda} (x)\right) ^ {2} \right] ^ {\frac {1}{2}} \left[ \mathbb {P} \{x \notin \Omega_ {t} \} \right] ^ {\frac {1}{2}} \\ \leq M _ {\alpha} \lambda^ {- \alpha / 2} \lambda^ {s / 2} t ^ {- q / 2} \| f _ {\rho} ^ {*} \| _ {L ^ {q}} ^ {q / 2}. \\ \end{array}
$$
Consequently, if  $l > \frac{\alpha\theta}{q}$ , then
$$
\left| \left| \mathbb {E} \xi (x) \mathbf {1} _ {\{x \notin \Omega_ {t} \}} \right| \right| _ {\mathcal {H}} = o (\lambda^ {s / 2}).
$$
Finally, the three requirements of  $l$  are
$$
l <   1 - \frac {\alpha + s}{2} \theta , \quad l > \frac {1}{q}, \quad \text {a n d} \quad l > \frac {\theta \alpha}{q},
$$
where  $q = \frac{2\alpha}{\alpha - s}$ . Since  $\theta < \beta = \alpha_0^{-1}$ , we can choose  $\alpha$  sufficiently close to  $\alpha_0$  such that  $\theta \alpha < 1$ . Then,
$$
(1 - \frac {\alpha + s}{2} \theta) - \frac {1}{q} = (1 - \theta \alpha) \left(\frac {\alpha + s}{2 \alpha}\right) > 0,
$$
$$
\frac {\theta \alpha}{q} <   \frac {1}{q} <   1 - \frac {\alpha + s}{2} \theta ,
$$
showing that we can choose some  $l$  satisfying all the requirements and the proof is finish.
A.4 The noiseless case
The case when  $\lambda = n^{-\theta}$  for  $\theta < \beta$  is already covered in Theorem A.2. For the case  $\theta \geq \beta$ , the approximation Lemma B.5 no longer holds, and we must reduce it to the former case. However, there is no direct monotone property of  $\mathbf{Bias}(\lambda)$ . Nevertheless, we have the following monotone relation about the operator norms, whose proof utilizes the idea in Lin et al. [2021, Proposition 6.1] with modification.
Proposition A.11. Let  $\psi_{\lambda} = \lambda (T_X + \lambda)^{-1}\in \mathcal{B}(\mathcal{H})$  . Suppose  $\lambda_1\leq \lambda_2$  , then for any  $s,p\geq 0$
$$
\left\| T ^ {s} \psi_ {\lambda_ {1}} ^ {p} \right\| _ {\mathcal {B} (\mathcal {H})} = \left\| \psi_ {\lambda_ {1}} ^ {p} T ^ {s} \right\| _ {\mathcal {B} (\mathcal {H})} \leq \left\| T ^ {s} \psi_ {\lambda_ {2}} ^ {p} \right\| _ {\mathcal {B} (\mathcal {H})} = \left\| \psi_ {\lambda_ {2}} ^ {p} T ^ {s} \right\| _ {\mathcal {B} (\mathcal {H})}. \tag {54}
$$
Proof. Let us denote by  $\preceq$  the partial order induced by positive operators. Since the function  $\lambda \mapsto \frac{\lambda}{z + \lambda}$  is monotone decreasing with  $\lambda$ , we obtain  $\psi_{\lambda_1}^{2p} \preceq \psi_{\lambda_2}^{2p}$ , which further implies
$$
T ^ {s} \psi_ {\lambda_ {1}} ^ {2 p} T ^ {s} \preceq T ^ {s} \psi_ {\lambda_ {2}} ^ {2 p} T ^ {s}.
$$
Then, since  $\| A\| ^2 = \| AA^*\|$ , we have
$$
\left\| T ^ {s} \psi_ {\lambda_ {1}} ^ {p} \right\| _ {\mathcal {B} (\mathcal {H})} ^ {2} = \left\| T ^ {s} \psi_ {\lambda_ {1}} ^ {2 p} T ^ {s} \right\| _ {\mathcal {B} (\mathcal {H})} \leq \left\| T ^ {s} \psi_ {\lambda_ {2}} ^ {2 p} T ^ {s} \right\| _ {\mathcal {B} (\mathcal {H})} = \left\| T ^ {s} \psi_ {\lambda_ {2}} ^ {p} \right\| _ {\mathcal {B} (\mathcal {H})} ^ {2},
$$
and the equality in (54) is proven by  $\| A\| = \| A^{*}\|$
Lemma A.12. Under Assumption 1,2,3, assume further  $s > 1$ . Suppose  $\lambda \asymp n^{-\theta}$  for  $\theta \geq \beta$ . Then,
$$
\operatorname {B i a s} ^ {2} (\lambda) = O _ {\mathbb {P}} ^ {\text {p o l y}} \left(n ^ {- \min  (s, 2) \beta}\right). \tag {55}
$$
Proof. Since  $f_{\rho}^{*}$  is given in (12) and  $s > 1$ , we have  $f_{\rho}^{*} \in [\mathcal{H}]^{t}$  for  $1 \leq t < s$ . In particular,  $f_{\rho}^{*} \in \mathcal{H}$  so the bias term can also be written as
$$
\operatorname {B i a s} (\lambda) = \left\| \lambda \left(T _ {X} + \lambda\right) ^ {- 1} f _ {\rho} ^ {*} \right\| _ {L ^ {2}}. \tag {56}
$$
Moreover, from the construction (8) of  $[\mathcal{H}]^t$ , we may assume  $f_{\rho}^{*} = T^{t / 2}g$  for some  $g\in L^2$  with  $\| g\|_{L^2}\leq C$ , and restrict further that  $t\leq 2$ . Let  $\tilde{\lambda}\asymp n^{-l}$  for  $l\in (0,\beta)$ . Then, using the same notation in Proposition A.11, we have
$$
\begin{array}{l} \mathbf {B i a s} (\lambda) = \left\| \psi_ {\lambda} f _ {\rho} ^ {*} \right\| _ {L ^ {2}} = \left\| T ^ {1 / 2} \psi_ {\lambda} T ^ {\frac {t - 1}{2}} \cdot T ^ {1 / 2} g \right\| _ {\mathcal {H}} \\ \leq \left\| T ^ {1 / 2} \psi_ {\lambda} T ^ {(t - 1) / 2} \right\| \cdot \left\| T ^ {1 / 2} g \right\| _ {\mathcal {H}} \\ \leq C \left\| T ^ {1 / 2} \psi_ {\lambda} ^ {1 / 2} \right\| \cdot \left\| \psi_ {\lambda} ^ {1 / 2} T ^ {\frac {t - 1}{2}} \right\| \\ \leq C \left\| T ^ {1 / 2} \psi_ {\tilde {\lambda}} ^ {1 / 2} \right\| \cdot \left\| \psi_ {\tilde {\lambda}} ^ {1 / 2} T ^ {\frac {t - 1}{2}} \right\| \\ \leq C \left\| T ^ {1 / 2} \psi_ {\tilde {\lambda}} ^ {1 / 2} \right\| \cdot \left\| \psi_ {\tilde {\lambda}} ^ {(2 - t) / 2} \right\| \cdot \left\| \psi_ {\tilde {\lambda}} ^ {\frac {t - 1}{2}} T ^ {\frac {t - 1}{2}} \right\| \\ = C \tilde {\lambda} ^ {t / 2} \left\| \psi_ {\tilde {\lambda}} ^ {(2 - t) / 2} \right\| \cdot \left\| T ^ {1 / 2} T _ {X \tilde {\lambda}} ^ {- 1 / 2} \right\| \cdot \left\| T ^ {\frac {t - 1}{2}} T _ {X \tilde {\lambda}} ^ {- \frac {t - 1}{2}} \right\| \\ \leq C \tilde {\lambda} ^ {t / 2} \left\| T ^ {1 / 2} T _ {X \tilde {\lambda}} ^ {- 1 / 2} \right\| ^ {t}, \\ \end{array}
$$
where we use Lemma B.6 for the last inequality. Finally, since  $\tilde{\lambda} \asymp n^{-l}$  for  $l < \beta$ , Lemma B.5 implies that with high probability we have
$$
\left\| T ^ {\frac {1}{2}} T _ {X \tilde {\lambda}} ^ {- \frac {1}{2}} \right\| = \left\| T ^ {\frac {1}{2}} T _ {\lambda} ^ {- \frac {1}{2}} T _ {\lambda} ^ {\frac {1}{2}} T _ {X \tilde {\lambda}} ^ {- \frac {1}{2}} \right\| \leq \left\| T ^ {\frac {1}{2}} T _ {\lambda} ^ {- \frac {1}{2}} \right\| \left\| T _ {\lambda} ^ {\frac {1}{2}} T _ {X \tilde {\lambda}} ^ {- \frac {1}{2}} \right\| \leq 1 \cdot \sqrt {3} = \sqrt {3}.
$$
$$
\mathbf {B i a s} (\lambda) = O _ {\mathbb {P}} \left(\tilde {\lambda} ^ {t / 2}\right) = O _ {\mathbb {P}} \left(n ^ {- t l / 2}\right).
$$
Since  $t < \min(s, 2)$  and  $l < \beta$  can arbitrarily close, we conclude (55).
Proof of Proposition 4.4. Let us denote  $\mathcal{F} = \left\{f:\| f_{\rho}^{*}\|_{[\mathcal{H}]^{s}}\leq R\right\}$  for convenience. Since  $f_{\rho}^{*}\in \mathcal{H}$ , the bias term can be given by
$$
\mathbf {B i a s} (\lambda) = \left\| f _ {\rho} ^ {*} - T _ {X} (T _ {X} + \lambda) ^ {- 1} f _ {\rho} ^ {*} \right\| _ {L ^ {2}} = \left\| (I - L _ {X}) f _ {\rho} ^ {*} \right\| _ {L ^ {2}}
$$
for a linear operator  $L_{X} = T_{X}(T_{X} + \lambda)^{-1}$  on  $\mathcal{H}$ . Then,
$$
\begin{array}{l} \sup  _ {f _ {\rho} ^ {*} \in \mathcal {F}} \mathbf {B i a s} (\lambda) = \sup  _ {f _ {\rho} ^ {*} \in \mathcal {F}} \left\| (I - L _ {X}) f _ {\rho} ^ {*} \right\| _ {L ^ {2}} \stackrel {(a)} {=} \sup  _ {\| g \| _ {\mathcal {H}} \leq R} \left\| T ^ {\frac {1}{2}} (I - L _ {X}) T ^ {\frac {s - 1}{2}} g \right\| _ {\mathcal {H}} \\ = \sup  _ {\| g \| _ {\mathcal {H}} \leq R} \left\| \left(T ^ {\frac {s}{2}} - T ^ {\frac {s}{2}} L _ {X} T ^ {\frac {s - 1}{2}}\right) g \right\| _ {\mathcal {H}} \\ = \left\| T ^ {\frac {s}{2}} - T ^ {\frac {s}{2}} L _ {X} T ^ {\frac {s - 1}{2}} \right\| _ {\mathscr {B} (\mathcal {H})} \\ \stackrel {(b)} {\geq} \lambda_ {n + 1} ^ {s / 2} = (n + 1) ^ {- s \beta / 2} = \Omega (n ^ {- s \beta / 2}), \\ \end{array}
$$
where in (a) we use the relation between the interpolation spaces and in (b) we use the fact that and  $\| A - B \| \geq \lambda_{n + 1}(A)$  for any operator  $B$  with rank at most  $n$  (see, for example, Simon [2015, Section 3.5]).
B Auxiliary results
$$
f (z) = \frac {z ^ {\alpha}}{z + \lambda}.
$$
(1) If  $\alpha = 0$ , then  $f(z)$  is monotone decreasing.
(2) If  $\alpha \in (0,1)$ , then  $f(z)$  is monotone increasing in  $[0,\frac{\alpha\lambda}{1 - \alpha}]$ , and decreasing in  $[\frac{\alpha\lambda}{1 - \alpha}, +\infty)$ . Consequently,  $f(z) \leq \lambda^{\alpha - 1}$ .
(3) If  $\alpha \geq 1$ , then  $f(z)$  monotone increasing on  $[0, +\infty)$ .
Proof. We simply notice that
$$
f ^ {\prime} (z) = \frac {z ^ {\alpha - 1}}{(z + \lambda) ^ {2}} (\alpha \lambda - (1 - \alpha) z).
$$
Proposition B.2. Suppose  $c_{\beta}i^{-\beta}\leq \lambda_i\leq C_\beta i^{-\beta}$  and  $p > 0$  , then as  $\lambda \to 0$  , we have
$$
\sum_ {i = 1} ^ {\infty} \left(\frac {\lambda_ {i} ^ {p}}{\lambda_ {i} + \lambda}\right) ^ {2} i ^ {- 1} \asymp \left\{ \begin{array}{l l} \lambda^ {2 (p - 1)}, & p <   1; \\ \ln \frac {1}{\lambda}, & p = 1; \\ 1, & p > 1. \end{array} \right. \tag {57}
$$
Proof. We first consider the case when  $p < 1$ . Since  $c_{\beta}i^{-\beta} \leq \lambda_i \leq C_\beta i^{-\beta}$ , from Proposition B.1, letting  $q = \frac{p}{1 - p}$ , we have
$$
\frac {\lambda_ {i} ^ {p}}{\lambda_ {i} + \lambda} \leq \left\{ \begin{array}{l l} \frac {C _ {\beta} ^ {p} i ^ {- p \beta}}{C _ {\beta} i ^ {- \beta} + \lambda}, & \quad \text {i f} \quad C _ {\beta} i ^ {- \beta} \leq q \lambda ; \\ \lambda_ {i} ^ {p} / \lambda_ {i} \leq C _ {\beta} ^ {p} i ^ {- (p - 1) \beta}, & \quad \text {i f} \quad C _ {\beta} i ^ {- \beta} > q \lambda ; \end{array} \right.
$$
$$
\begin{array}{l} \sum_ {i = 1} ^ {\infty} \left(\frac {\lambda_ {i} ^ {p}}{\lambda_ {i} + \lambda}\right) ^ {2} i ^ {- 1} \leq C \sum_ {i: C _ {\beta} i ^ {- \beta} > q \lambda} i ^ {- 2 (p - 1) \beta - 1} + C \sum_ {i: C _ {\beta} i ^ {- \beta} \leq q \lambda} \frac {i ^ {- 2 p \beta}}{(C _ {\beta} i ^ {- \beta} + \lambda) ^ {2}} i ^ {- 1} \\ =: S _ {1} + S _ {2}. \\ \end{array}
$$
For  $S_{1}$ , noticing  $C_{\beta}i^{-\beta} > q\lambda$  implies  $i < (q\lambda / C_{\beta})^{-1/\beta}$ , we have
$$
S _ {1} \leq C \sum_ {i = 1} ^ {\lfloor (q \lambda / C _ {\beta}) ^ {- 1 / \beta} \rfloor} i ^ {- 2 (p - 1) \beta - 1} \leq C \lambda^ {2 (p - 1)}.
$$
For  $S_{2}$ , using Proposition B.1 again we have
$$
\begin{array}{l} S _ {2} \leq C \int_ {\left(q \lambda / C _ {\beta}\right) ^ {- 1 / \beta} - 1} ^ {\infty} \frac {x ^ {- 2 p \beta}}{\left(C _ {\beta} x ^ {- \beta} + \lambda\right) ^ {2}} x ^ {- 1} \mathrm {d} x \\ = C \lambda^ {2 p - 2} \int_ {(C _ {\beta} / q) ^ {1 / \beta} - \lambda^ {1 / \beta}} ^ {\infty} \frac {y ^ {- 2 p \beta}}{(C _ {\beta} y ^ {- \beta} + 1) ^ {2}} y ^ {- 1} \mathrm {d} y (x = \lambda^ {- 1 / \beta} y) \\ \leq C \lambda^ {2 p - 2}, \\ \end{array}
$$
where we note that the last integral is bounded above by a constant. Therefore, we conclude that  $\left\| f_{\lambda} - f_{\rho}^{*}\right\|_{[\mathcal{H}]^{\gamma}}^{2}\leq C\lambda^{2p - 2}$ . For the lower bound, if  $C_{\beta}i^{-\beta}\leq q\lambda$ , we have
$$
\frac {\lambda_ {i} ^ {p}}{\lambda_ {i} + \lambda} \geq \frac {c _ {\beta} ^ {p} i ^ {- p \beta}}{c _ {\beta} i ^ {- \beta} + \lambda},
$$
$$
\begin{array}{l} \sum_ {i = 1} ^ {\infty} \left(\frac {\lambda_ {i} ^ {p}}{\lambda_ {i} + \lambda}\right) ^ {2} i ^ {- 1} \geq C \int_ {(q \lambda / C _ {\beta}) ^ {- 1 / \beta}} ^ {\infty} \frac {x ^ {- 2 p \beta}}{(C _ {\beta} x ^ {- \beta} + \lambda) ^ {2}} x ^ {- 1} \mathrm {d} x \\ = C \lambda^ {2 p - 2} \int_ {(C _ {\beta} / p) ^ {1 / \beta}} ^ {\infty} \frac {y ^ {- 2 p \beta}}{(C _ {\beta} y ^ {- \beta} + 1) ^ {2}} y ^ {- 1} d y \\ \geq C \lambda^ {2 p - 2}, \\ \end{array}
$$
where we note that the last integral is independent of  $\lambda$ .
For the case  $p = 1$ , by Proposition B.1, we have
$$
\begin{array}{l} \sum_ {i = 1} ^ {\infty} \left(\frac {\lambda_ {i}}{\lambda_ {i} + \lambda}\right) ^ {2} i ^ {- 1} \leq C \sum_ {i = 1} ^ {\infty} \left(\frac {i ^ {- \beta}}{C _ {\beta} i ^ {- \beta} + \lambda}\right) ^ {2} i ^ {- 1} \\ \leq C \sum_ {i = 1} ^ {\lfloor 2 \lambda^ {- 1 / \beta} \rfloor} \left(\frac {i ^ {- \beta}}{C _ {\beta} i ^ {- \beta} + \lambda}\right) ^ {2} i ^ {- 1} + C \sum_ {i = \lfloor 2 \lambda^ {- 1 / \beta} \rfloor + 1} ^ {\infty} \left(\frac {i ^ {- \beta}}{C _ {\beta} i ^ {- \beta} + \lambda}\right) ^ {2} i ^ {- 1} \\ \leq C \sum_ {i = 1} ^ {\lfloor 2 \lambda^ {- 1 / \beta} \rfloor} i ^ {- 1} + C \int_ {2 \lambda^ {- 1 / \beta}} ^ {\infty} \left(\frac {x ^ {- \beta}}{C _ {\beta} x ^ {- \beta} + \lambda}\right) ^ {2} x ^ {- 1} d x \\ \leq C \ln \frac {1}{\lambda} + C \int_ {2} ^ {\infty} \left(\frac {y ^ {- \beta}}{C _ {\beta} y ^ {- \beta} + 1}\right) ^ {2} y ^ {- 1} d y \\ \leq C \ln \frac {1}{\lambda}. \\ \end{array}
$$
For the lower bound, we have
$$
\begin{array}{l} \sum_ {i = 1} ^ {\infty} \left(\frac {\lambda_ {i}}{\lambda_ {i} + \lambda}\right) ^ {2} i ^ {- 1} \geq c \sum_ {i = 1} ^ {\lfloor \lambda^ {- 1 / \beta} \rfloor} \left(\frac {i ^ {- \beta}}{c _ {\beta} i ^ {- \beta} + \lambda}\right) ^ {2} i ^ {- 1} \\ \geq c \sum_ {i = 1} ^ {\lfloor \lambda^ {- 1 / \beta} \rfloor} i ^ {- 1} \geq c \ln \frac {1}{\lambda}. \\ \end{array}
$$
For the case  $p > 1$ , by Proposition B.1, we have
$$
\sum_ {i = 1} ^ {\infty} \left(\frac {\lambda_ {i} ^ {p}}{\lambda_ {i} + \lambda}\right) ^ {2} i ^ {- 1} \leq C \sum_ {i = 1} ^ {\infty} \frac {i ^ {- 2 p \beta}}{(C _ {\beta} i ^ {- \beta} + \lambda) ^ {2}} i ^ {- 1} \leq C \sum_ {i = 1} ^ {\infty} i ^ {- 2 (p - 1) \beta - 1} \leq C,
$$
since the last series is summable. The lower bound is derived by
$$
\sum_ {i = 1} ^ {\infty} \left(\frac {\lambda_ {i} ^ {p}}{\lambda_ {i} + \lambda}\right) ^ {2} i ^ {- 1} \geq \frac {\lambda_ {1} ^ {p}}{\lambda_ {1} + \lambda} \geq c.
$$
Proposition B.3. Under Assumption 1, for any  $p \geq 1$ , we have
$$
\mathcal {N} _ {p} (\lambda) = \operatorname {t r} \left(T T _ {\lambda} ^ {- 1}\right) ^ {p} = \sum_ {i = 1} ^ {\infty} \left(\frac {\lambda_ {i}}{\lambda + \lambda_ {i}}\right) ^ {p} \asymp \lambda^ {- 1 / \beta}. \tag {58}
$$
Proof. Since  $c i^{-\beta} \leq \lambda_i \leq C i^{-\beta}$ , we have
$$
\begin{array}{l} \mathcal {N} _ {p} (\lambda) = \sum_ {i = 1} ^ {\infty} \left(\frac {\lambda_ {i}}{\lambda_ {i} + \lambda}\right) ^ {p} \leq \sum_ {i = 1} ^ {\infty} \left(\frac {C i ^ {- \beta}}{C i ^ {- \beta} + \lambda}\right) ^ {p} = \sum_ {i = 1} ^ {\infty} \left(\frac {C}{C + \lambda i ^ {\beta}}\right) ^ {p} \\ \leq \int_ {0} ^ {\infty} \left(\frac {C}{\lambda x ^ {\beta} + C}\right) ^ {p} d x = \lambda^ {- 1 / \beta} \int_ {0} ^ {\infty} \left(\frac {C}{y ^ {\beta} + C}\right) ^ {p} d y \leq \tilde {C} \lambda^ {- 1 / \beta}. \\ \end{array}
$$
for some constant  $C$ . The lower bound is similar.
The following inequality about vector-valued random variables is well-known in the literature [Capon-netto and De Vito, 2007].
Lemma B.4. Let  $H$  be a real separable Hilbert space. Let  $\xi, \xi_1, \ldots, \xi_n$  be i.i.d. random variables taking values in  $H$ . Assume that
$$
\mathbb {E} \| \xi - \mathbb {E} \xi \| _ {H} ^ {m} \leq \frac {1}{2} m! \sigma^ {2} L ^ {m - 2}, \quad \forall m = 2, 3, \dots . \tag {59}
$$
Then for fixed  $\delta \in (0,1)$ , one has
$$
\mathbb {P} \left\{\left\| \frac {1}{n} \sum_ {i = 1} ^ {n} \xi_ {i} - \mathbb {E} \xi \right\| _ {H} \leq 2 \left(\frac {L}{n} + \frac {\sigma}{\sqrt {n}}\right) \ln \frac {2}{\delta} \right\} \geq 1 - \delta . \tag {60}
$$
Particularly, a sufficient condition for (59) is
$$
\left\| \xi \right\| _ {H} \leq \frac {L}{2} a. s., a n d \mathbb {E} \| \xi \| _ {H} ^ {2} \leq \sigma^ {2}.
$$