SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Let us now consider the case where  $\| \theta \| \leq D$ , and  $\theta$  is supported on the top-  $k$  eigenvectors of  $K$ , we get  $\| f^{*} \| = \left\| \Sigma^{-1/2} \theta \right\| \leq D \lambda_{k}^{-1/2}$  where  $\lambda_{k}$  is the  $k$ -th eigenvalue of  $T$ . Hence, considering an RKHS  $\mathcal{F}$ , under Assumption 2, for any algorithm  $\mathcal{A}$ , there exists a function such that  $\| f^{*} \|_{\mathcal{F}} \leq D$ , and
$$
\mathbb {E} _ {\mathcal {D} _ {n}} \left[ \mathcal {E} (\mathcal {A} (\mathcal {D} _ {n})) \right] \geq \frac {1}{6} \max  _ {k \in \mathbb {N}} \min  \left\{\frac {\varepsilon^ {2} k}{n}, D ^ {2} \lambda_ {k} \right\},
$$
Considering  $\lambda_{k} = k^{-2\alpha /d}$  and  $k = (nD^{2} / \varepsilon^{2})^{2\alpha /(2\alpha +d)}$  leads to the result.
In terms of implications of Theorem 7, in harmonics settings, it is possible to build an integral operator such that the eigenfunctions of  $K$  are known to be regular smooth functions, e.g., the trigonometric polynomials  $f_{m}:x\to \exp (-im^{\top}x)$  for  $m\in \mathbb{Z}^d$  ordered according to  $\| m\| = \omega$ . As such, for  $m\in \mathbb{Z}^d$  with  $\| m\| = \omega$ ,  $f_{m}$  is expected to be ordered around the  $\omega^d$ -th eigenfunctions of  $K$ , typically associated with the eigenvalue  $\lambda_{\omega^d}\approx (\omega^d)^{-2\alpha /d} = \omega^{-2\alpha}$  if  $\mathcal{F}$  corresponds to  $H^{\alpha}$ . Using that when  $f^{*}$  is the  $k$ -th eigenfunctions of  $K$ , we have  $\| f^{*}\| = \lambda_{k}^{-1}$ , Theorem 3 would become that for any algorithm  $\mathcal{A}$ , under Assumption 2, there exists a function  $f^{*}$  such that  $\| f^{*}\|_{\mathcal{F}}\leq \| f_m\|_{\mathcal{F}}$  with  $\| m\| = \omega$  and,
$$
\mathbb {E} _ {\mathcal {D} _ {n}} \left[ \mathcal {E} (\mathcal {A} (\mathcal {D} _ {n})) \right] \gtrsim \frac {\varepsilon^ {4 \alpha / (2 \alpha + d)} \omega^ {2 d \alpha / (2 \alpha + d)}}{n ^ {2 \alpha / (2 \alpha + d)}}.
$$
When  $\alpha$  scales linearly with  $d$  (so to beat the curse of dimensionality), this lower bound presents a constant that is exponential with  $d$ .
A.3 Precise Excess Risk Bound
This subsection is devoted to the proof of the bound in Theorem 4.
For ease of notation, we will use the finite-dimensional notation  $u^{\top}w$  to denote the inner product  $\langle u,v\rangle$  in (infinite-dimensional) Hilbert spaces. Moreover, we will simply write  $\| \cdot \|$  for both  $\| \cdot \|_{\mathcal{H}}$
and the operator norm on  $\mathcal{H}$  (depending on context),  $L^2$  for  $L^2 (\rho_{\mathcal{X}})$ , and  $\| \cdot \| _2$  for both  $\| \cdot \|_{L^2}$  and the operator norm on  $L^2$ .
For the sake of clarity, we have expressed all our statements in terms of operators on  $L^2$ . For the proofs, it is more convenient to work in  $\mathcal{H}$ . Let us introduce the embedding
$$
S: \mathcal {H} \to L ^ {2}, \quad \theta \mapsto (x \mapsto \theta^ {\top} \varphi (x)).
$$
From  $S$ , one can take its adjoint  $S^*$  and check that  $K = SS^*$ .  $K$  is isometric to the (non-centered) covariance operator
$$
\Sigma = S ^ {*} S = \mathbb {E} [ \varphi (X) \otimes \varphi (X) ].
$$
$$
\left\| S \theta \right\| _ {2} ^ {2} = \mathbb {E} \left[ (\varphi (X) ^ {\top} \theta) ^ {2} \right] \leq \mathbb {E} \left[ \| \varphi (X) \| ^ {2} \| \theta \| ^ {2} \right] = \| \varphi \| _ {2} ^ {2} \| \theta \| ^ {2},
$$
which implies that  $K$  is a continuous operator as soon as  $\varphi \in L^2$ . The kernel ridge regression estimator (8) is characterized as
$$
f _ {n} = S \left(\Sigma_ {n} + 1\right) ^ {- 1} S _ {n} ^ {*} \mathbb {Y},
$$
$$
S _ {n}: \mathcal {H} \to \mathbb {R} ^ {n}, \theta \mapsto (\theta^ {\top} \varphi (X _ {i})) _ {i \in [ n ]}, \qquad \Sigma_ {n} = S _ {n} ^ {*} S _ {n}, \qquad \mathbb {Y} = (Y _ {i}) _ {i \in [ n ]} \in \mathbb {R} ^ {n}.
$$
Endowing  $\mathbb{R}^n$  with the scalar product  $\langle a,b\rangle = \frac{1}{n}\sum_{i\in [n]}a_ib_i$ , we have
$$
S _ {n} ^ {*} \mathbb {Y} = \frac {1}{n} \sum_ {i \in [ n ]} Y _ {i} \varphi (X _ {i}), \qquad \Sigma_ {n} = \frac {1}{n} \sum_ {i = 1} ^ {n} \varphi (X _ {i}) \otimes \varphi (X _ {i}).
$$
It is useful to define  $\varepsilon_{i}$  as the difference between  $Y_{i}$  and  $f^{*}(X_{i}) = \mathbb{E}[Y_{i} \mid X = X_{i}]$ , which can be seen as the labeling noise and average to zero. For simplicity, we will first assume that our model is well-specified, i.e.  $f^{*} = S\theta_{*}$  for some  $\theta^{*}$ . We have, with  $E = (\varepsilon_{i})_{i \in [n]} \in \mathbb{R}^{n}$ ,
$$
Y _ {i} = f ^ {*} \left(X _ {i}\right) + \varepsilon_ {i} = \varphi \left(X _ {i}\right) ^ {\top} \theta_ {*} + \varepsilon_ {i}, \quad \mathbb {Y} = S _ {n} \theta_ {*} + E.
$$
$$
f _ {n} = S \left(\Sigma_ {n} + 1\right) ^ {- 1} \Sigma_ {n} \theta_ {*} + S \left(\Sigma_ {n} + 1\right) ^ {- 1} S _ {n} ^ {*} E.
$$
Let  $\mathbb{X} = (X_i)_{i\in [n]}$ . we have
$$
\begin{array}{l} \mathbb {E} _ {\mathcal {D} _ {n}} \left[ \mathcal {E} (f _ {n}) \mid \mathbb {X} \right] = \mathbb {E} _ {\mathcal {D} _ {n}} \left[ \| f _ {n} - f ^ {*} \| _ {2} ^ {2} \mid \mathbb {X} \right] \\ = \left\| S \left(\Sigma_ {n} + 1\right) ^ {- 1} \Sigma_ {n} \theta_ {*} - S \theta_ {*} \right\| _ {2} ^ {2} \\ + \mathbb {E} _ {\mathcal {D} _ {n}} \left[ \left\| S (\Sigma_ {n} + 1) ^ {- 1} S _ {n} ^ {*} E \right\| _ {2} ^ {2} \mid \mathbb {X} \right] \\ + 2 \mathbb {E} _ {\mathcal {D} _ {n}} \left[ \left(S (\Sigma_ {n} + 1) ^ {- 1} \Sigma_ {n} \theta_ {*} - S \theta_ {*}\right) ^ {\top} S (\Sigma_ {n} + 1) ^ {- 1} S _ {n} ^ {*} E \mid \mathbb {X} \right] \\ = \left\| S (\Sigma_ {n} + 1) ^ {- 1} \theta_ {*} \right\| _ {2} ^ {2} \\ + \mathbb {E} _ {\mathcal {D} _ {n}} \left[ \left\| S (\Sigma_ {n} + 1) ^ {- 1} S _ {n} ^ {*} E \right\| _ {2} ^ {2} \mid \mathbb {X} \right] \\ + 2 \left(S _ {n} \left(\Sigma_ {n} + 1\right) ^ {- 1} \Sigma \left(\Sigma_ {n} + 1\right) ^ {- 1} \theta_ {*}\right) ^ {\top} \mathbb {E} _ {\mathcal {D} _ {n}} [ E | \mathbb {X} ] \\ = \left\| S (\Sigma_ {n} + 1) ^ {- 1} \theta_ {*} \right\| _ {2} ^ {2} + \mathbb {E} _ {\mathcal {D} _ {n}} \left[ \left\| S (\Sigma_ {n} + 1) ^ {- 1} S _ {n} ^ {*} E \right\| _ {2} ^ {2} \mid \mathbb {X} \right], \\ \end{array}
$$
where in the third equality we used  $I - (\Sigma_n + 1)^{-1}\Sigma_n = (\Sigma_n + 1)^{-1}$ , and in the last one that  $\mathbb{E}_{\mathcal{D}_n}[E|\mathbb{X}] = 0$ . Assuming for simplicity that the noise is homoscedastic so that  $\mathbb{E}[EE^{\top}] = \varepsilon^{2}I$  for  $\varepsilon > 0$ , we obtain
$$
\begin{array}{l} \mathbb {E} _ {\mathcal {D} _ {n}} \left[ \left\| S (\Sigma_ {n} + 1) ^ {- 1} S _ {n} ^ {*} E \right\| _ {2} ^ {2} \mid \mathbb {X} \right] = \frac {1}{n} \operatorname {T r} \left(S _ {n} (\Sigma_ {n} + 1) ^ {- 1} \Sigma (\Sigma_ {n} + 1) ^ {- 1} S _ {n} ^ {*} \mathbb {E} _ {\mathcal {D} _ {n}} \left[ E E ^ {\top} \mid \mathbb {X} \right]\right) \\ = \frac {\varepsilon^ {2}}{n} \operatorname {T r} \left(\Sigma (\Sigma_ {n} + 1) ^ {- 2} \Sigma_ {n}\right), \\ \end{array}
$$
where the  $1 / n$  factor arises from the fact that  $E^{*} = E^{\top} / n$  in the geometry we have considered on  $\mathbb{R}^n$ . Finally we have retrieved the following standard bias-variance decomposition result.
Lemma 2 (Bias-Variance decomposition). When our model is well-specified so that  $f^{*} = S\theta_{*}$  with  $\theta_{*}\in \mathcal{H}$ , and when the noise in the label is homoscedastic with variance  $\varepsilon^2$ , the estimator (8) verifies
$$
\mathbb {E} _ {\mathcal {D} _ {n}} \left[ \mathcal {E} \left(f _ {n}\right) \mid \mathbb {X} \right] = \underbrace {\left\| S \left(\Sigma_ {n} + 1\right) ^ {- 1} \theta_ {*} \right\| _ {2} ^ {2}} _ {\mathcal {B} _ {n}} + \frac {\varepsilon^ {2}}{n} \underbrace {\operatorname {T r} \left(\Sigma \left(\Sigma_ {n} + 1\right) ^ {- 2} \Sigma_ {n}\right)} _ {\nu_ {n}}. \tag {18}
$$
We would like to get the limit when  $n$  goes to infinity in equation (18). We expect the first term to concentrate towards  $\left\| S(\Sigma + 1)^{-1}\theta_{*}\right\|_{2}^{2}$ , and the second term to  $\mathrm{Tr}(\Sigma^2 (\Sigma + 1)^{-2})$ .
A.3.1 Bounding the bias term
Let us begin by working out the term  $\mathcal{B}_n = \left\| S(\Sigma_n + 1)^{-1}\theta_*\right\|_2^2$ . We first introduce some notation to make derivations shorter. Let  $E_{n} = \Sigma_{n} - \Sigma$ ,  $\Sigma_{1} = \Sigma +1$  and  $F_{n} = -\Sigma_{1}^{-1 / 2}E_{n}\Sigma_{1}^{-1 / 2}$ . As long as  $\| F_n\| < 1$ , we have
$$
\begin{array}{l} \mathcal {B} _ {n} = \theta_ {*} ^ {\top} (\Sigma_ {n} + 1) ^ {- 1} \Sigma (\Sigma_ {n} + 1) ^ {- 1} \theta_ {*} \\ = \theta_ {*} ^ {\top} \left(\Sigma_ {1} + E _ {n}\right) ^ {- 1} \Sigma \left(\Sigma_ {1} + E _ {n}\right) ^ {- 1} \theta_ {*} \\ = \theta_ {*} ^ {\top} \Sigma_ {1} ^ {- 1 / 2} (I - F _ {n}) ^ {- 1} \Sigma_ {1} ^ {- 1} \Sigma (I - F _ {n}) ^ {- 1} \Sigma_ {1} ^ {- 1 / 2} \theta_ {*} \\ = \sum_ {i, j \in \mathbb {N}} \theta_ {*} ^ {\top} \Sigma_ {1} ^ {- 1 / 2} F _ {n} ^ {i} \Sigma_ {1} ^ {- 1} \Sigma F _ {n} ^ {j} \Sigma_ {1} ^ {- 1 / 2} \theta_ {*}. \\ \end{array}
$$
Let us assume for a moment that
$$
\left\| \left(\Sigma \Sigma_ {1} ^ {- 1}\right) ^ {- 1 / 2} F _ {n} \left(\Sigma \Sigma_ {1} ^ {- 1}\right) ^ {1 / 2} \right\| \leq \| F _ {n} \|. \tag {19}
$$
If equation (19) holds, then
$$
\begin{array}{l} \left| \mathcal {B} _ {n} - \theta_ {*} ^ {\top} \Sigma_ {1} ^ {- 2} \Sigma \theta_ {*} \right| = \left| \sum_ {i, j \in \mathbb {N}; i + j \neq 0} \theta_ {*} ^ {\top} \Sigma_ {1} ^ {- 1 / 2} F _ {n} ^ {i} \Sigma_ {1} ^ {- 1} \Sigma F _ {n} ^ {j} \Sigma_ {1} ^ {- 1 / 2} \theta_ {*} \right| \\ \leq \sum_ {i + j \neq 0} \left| \theta_ {*} ^ {\top} \Sigma_ {1} ^ {- 1 / 2} F _ {n} ^ {i} \Sigma_ {1} ^ {- 1} \Sigma F _ {n} ^ {j} \Sigma_ {1} ^ {- 1 / 2} \theta_ {*} \right| \\ = \sum_ {i + j \neq 0} \left| \left\langle \Sigma_ {1} ^ {- 1 / 2} \left(\Sigma_ {1} ^ {- 1} \Sigma\right) ^ {1 / 2} \theta_ {*}, \left(\left(\Sigma_ {1} ^ {- 1} \Sigma\right) ^ {- 1 / 2} F _ {n} ^ {i} \Sigma_ {1} ^ {- 1} \Sigma F _ {n} ^ {j} \left(\Sigma_ {1} ^ {- 1} \Sigma\right) ^ {- 1 / 2}\right) \left(\Sigma_ {1} ^ {- 1} \Sigma\right) ^ {1 / 2} \Sigma_ {1} ^ {- 1 / 2} \theta_ {*} \right\rangle \right| \\ \leq \sum_ {i + j \neq 0} \left\| \Sigma_ {1} ^ {- 1 / 2} (\Sigma_ {1} ^ {- 1} \Sigma) ^ {1 / 2} \theta_ {*} \right\| ^ {2} \left\| (\Sigma_ {1} ^ {- 1} \Sigma) ^ {- 1 / 2} F _ {n} ^ {i} \Sigma_ {1} ^ {- 1} \Sigma F _ {n} ^ {j} (\Sigma_ {1} ^ {- 1} \Sigma) ^ {- 1 / 2} \right\| \\ = \theta_ {*} ^ {\top} \Sigma_ {1} ^ {- 2} \Sigma \theta_ {*} \sum_ {i + j \neq 0} \left\| \left(\left(\Sigma \Sigma_ {1} ^ {- 1}\right) ^ {- 1 / 2} F _ {n} \left(\Sigma \Sigma_ {1} ^ {- 1}\right) ^ {1 / 2}\right) ^ {i + j} \right\| \\ \leq \theta_ {*} ^ {\top} \Sigma_ {1} ^ {- 2} \Sigma \theta_ {*} \sum_ {i, j \in \mathbb {N}; i + j \neq 0} \left\| (\Sigma \Sigma_ {1} ^ {- 1}) ^ {- 1 / 2} F _ {n} (\Sigma \Sigma_ {1} ^ {- 1}) ^ {1 / 2} \right\| ^ {i + j} \\ = \theta_ {*} ^ {\top} \Sigma_ {1} ^ {- 2} \Sigma \theta_ {*} \sum_ {i \in \mathbb {N}} (i + 2) \left\| (\Sigma \Sigma_ {1} ^ {- 1}) ^ {- 1 / 2} F _ {n} (\Sigma \Sigma_ {1} ^ {- 1}) ^ {1 / 2} \right\| ^ {i + 1} \\ \leq \theta_ {*} ^ {\top} \Sigma_ {1} ^ {- 2} \Sigma \theta_ {*} \sum_ {i \in \mathbb {N}} (i + 2) \| F _ {n} \| ^ {i + 1} = \theta_ {*} ^ {\top} \Sigma_ {1} ^ {- 2} \Sigma \theta_ {*} \int_ {0} ^ {\infty} (\lfloor x \rfloor + 2) \| F _ {n} \| ^ {\lceil x \rceil} d x \\ \leq \theta_ {*} ^ {\top} \Sigma_ {1} ^ {- 2} \Sigma \theta_ {*} \int_ {0} ^ {\infty} (x + 2) \| F _ {n} \| ^ {x} d x = \theta_ {*} ^ {\top} \Sigma_ {1} ^ {- 2} \Sigma \theta_ {*} \frac {1 - 2 \log (\| F _ {n} \|)}{\log^ {2} (\| F _ {n} \|)}. \\ \end{array}
$$
This inequality is useful as long as  $\| F_n\|$  is small enough, which is not always true. When  $\| F_n\|$  is large, we can instead proceed with the simpler bound
$$
\left| \mathcal {B} _ {n} - \theta_ {*} ^ {\top} \Sigma_ {1} ^ {- 2} \Sigma \theta_ {*} \right| \leq \mathcal {B} _ {n} + \theta_ {*} ^ {\top} \Sigma_ {1} ^ {- 2} \Sigma \theta_ {*} \leq 2 \theta_ {*} ^ {\top} \Sigma \theta_ {*} = 2 \| f ^ {*} \| _ {2} ^ {2}.
$$
Therefore, rewriting the limit as
$$
\begin{array}{l} \theta_ {*} ^ {\top} (\Sigma + 1) ^ {- 2} \Sigma \theta_ {*} = \left(\Sigma^ {1 / 2} \theta_ {*}\right) ^ {\top} (\Sigma + 1) ^ {- 2} \Sigma^ {1 / 2} \theta_ {*} = \left(S \theta_ {*}\right) ^ {\top} (K + 1) ^ {- 2} S \theta_ {*} \\ = \left(f ^ {*}\right) ^ {\top} (K + 1) ^ {- 2} f ^ {*} = \left\| (K + 1) ^ {- 1} f ^ {*} \right\| _ {2} ^ {2} = \mathcal {S} (K), \\ \end{array}
$$
we split the bias error as
$$
\begin{array}{l} \left| \mathcal {B} _ {n} - \theta_ {*} ^ {\top} \Sigma_ {1} ^ {- 2} \Sigma \theta_ {*} \right| \leq 2 \| f ^ {*} \| _ {2} ^ {2} \mathbf {1} _ {\| F _ {n} \| > 1 / 2} + \mathcal {S} (K) \frac {1 - 2 \log (\| F _ {n} \|)}{\log^ {2} (\| F _ {n} \|)} \mathbf {1} _ {\| F _ {n} \| \leq 1 / 2} \\ \leq 2 \| f ^ {*} \| _ {2} ^ {2} \mathbf {1} _ {\| F _ {n} \| > 1 / 2} - \frac {3 S (K)}{\log (\| F _ {n} \|)} \mathbf {1} _ {\| F _ {n} \| \leq 1 / 2}. \\ \end{array}
$$
Taking the expectation and using the convexity of the absolute value, we obtain
$$
\left| \mathbb {E} _ {\mathcal {D} _ {n}} [ \mathcal {B} _ {n} ] - \theta_ {*} ^ {\top} \Sigma_ {1} ^ {- 2} \Sigma \theta_ {*} \right| \leq 2 \| f ^ {*} \| _ {2} ^ {2} \mathbb {P} (\| F _ {n} \| > 1 / 2) - \mathcal {S} (K) \int_ {0} ^ {1 / 2} \frac {3 \mathbb {P} (\| F _ {n} \| > x)}{\log (x)} \mathrm {d} x.
$$
We now proceed with an exponential concentration inequality on  $\| F_n\|$ . We will use the one of Cabannes et al. [5], Eq. (25). As long as  $1\leq \| K\|$ , we have
$$
\mathbb {P} \left(\| F _ {n} \| > t\right) \leq 2 8 \mathcal {N} _ {1} (K) \exp \left(- \frac {n t ^ {2}}{\mathcal {N} _ {+} (K) (1 + t)}\right).
$$
This inequality shows the restrictive notion of effective dimension, which is useful to ensure the good conditioning of the linear system implicitly encoded in (8), and, in essence, bound all moments of  $\varphi(X)$ . As long as  $\mathcal{N}_{+}(K) \leq 3n/2$ , we can compute the integral as
$$
\begin{array}{l} - \int_ {0} ^ {1 / 2} \frac {\mathbb {P} \left(\| F _ {n} \| > x\right)}{\log (x)} d x \leq - 2 8 \mathcal {N} _ {1} (K) \int_ {0} ^ {1 / 2} \frac {\exp (- 3 n x ^ {2} / 2 \mathcal {N} _ {+} (K))}{\log (x)} d x \\ = - 2 8 \mathcal {N} _ {1} (K) \frac {\mathcal {N} _ {+} ^ {1 / 2} (K)}{1 . 5 ^ {1 / 2} n ^ {1 / 2}} \int_ {0} ^ {1 / 2} \frac {\exp (- u ^ {2})}{\log (u) + \log (3 n / 2 \mathcal {N} _ {\infty} ^ {2} (K))} d u \\ \leq - 2 8 \frac {\mathcal {N} _ {1} (K) \mathcal {N} _ {+} ^ {1 / 2} (K)}{1 . 5 ^ {1 / 2} n ^ {1 / 2}} \int_ {0} ^ {1 / 2} \frac {\exp (- u ^ {2})}{\log (u)} \mathrm {d} u \leq \frac {8 \mathcal {N} _ {1} (K) \mathcal {N} _ {+} ^ {1 / 2} (K)}{n ^ {1 / 2}}. \\ \end{array}
$$
We recall that the bounds above were derived under condition (19), which is rather strong. However, the attentive reader would remark that a much laxer assumption is sufficient, which we introduce thereafter.
Assumption 3. There exists a constant  $c$  such that, for all  $i,j\in \mathbb{N}$
$$
\mathbb {E} \left[ \left\| \left(\Sigma \Sigma_ {1} ^ {- 1}\right) ^ {-. 5} F _ {n} ^ {i} \Sigma \Sigma_ {1} ^ {- 1} F _ {n} ^ {j} \left(\Sigma \Sigma_ {1} ^ {- 1}\right) ^ {-. 5} \right\| | \| F _ {n} \| \leq 1 / 2 \right] \leq c ^ {2} \mathbb {E} \left[ \| F _ {n} \| ^ {i + j} \mid \| F _ {n} \| \leq 1 / 2 \right]. \tag {20}
$$
Assumption 3 notably holds when  $\mathcal{F}$  is finite dimensional with  $c^2 = \left\| K^{-1}\right\|_2^{-1}(\| K\|_2 + 1)$ . As such all our lower-bound results can be cast with finite-dimensional approximation of infinite-dimensional RKHS. Under Assumption 3, we get
$$
\begin{array}{l} \mathbb {E} \left[ \left| \mathcal {B} _ {n} - \theta_ {*} ^ {\top} \Sigma_ {1} ^ {- 2} \Sigma \theta_ {*} \right| \| F _ {n} \| \leq 1 / 2 \right] \\ \leq \sum_ {i + j \neq 0} \left\| \Sigma_ {1} ^ {- 1 / 2} \left(\Sigma_ {1} ^ {- 1} \Sigma\right) ^ {1 / 2} \theta_ {*} \right\| ^ {2} \mathbb {E} \left[ \left\| \left(\Sigma_ {1} ^ {- 1} \Sigma\right) ^ {- 1 / 2} F _ {n} ^ {i} \Sigma_ {1} ^ {- 1} \Sigma F _ {n} ^ {j} \left(\Sigma_ {1} ^ {- 1} \Sigma\right) ^ {- 1 / 2} \right\| \right| \| F _ {n} \| \leq 1 / 2 \Bigg ] \\ \leq c ^ {2} \sum_ {i, j \in \mathbb {N}; i + j \neq 0} \left\| \Sigma_ {1} ^ {- 1 / 2} \left(\Sigma_ {1} ^ {- 1} \Sigma\right) ^ {1 / 2} \theta_ {*} \right\| ^ {2} \mathbb {E} \left[ \| F _ {n} \| ^ {i + j} \mid \| F _ {n} \| \leq 1 / 2 \right], \\ \end{array}
$$
which allows us to proceed with the precedent derivations without assuming that (19) holds.
While the previous results were achieved for  $f^{*} \in \mathcal{F}$ , they can be extended by density to any  $f^{*}$  in the closure of  $\mathcal{F}$  in  $L^2(\rho_{\mathcal{X}})$ , i.e.  $f \in (\ker K)^{\perp}$ , leading to the following result.
Proposition 3. When  $f^{*} \in (\ker K)^{\perp}$  and  $1 \leq \| \Sigma \|$ , under the technical Assumption 3, the bias term can be bounded from above and below by
$$
\left| \mathbb {E} _ {\mathcal {D} _ {n}} \left[ \mathcal {B} _ {n} \right] - \mathcal {S} (K) \right| \leq c ^ {2} \mathcal {N} _ {1} (K) \left(5 6 \| f ^ {*} \| _ {2} ^ {2} \exp \left(- \frac {n}{6 \mathcal {N} _ {+} (K)}\right) + \frac {8 \mathcal {S} (K) \mathcal {N} _ {+} ^ {1 / 2} (K)}{n ^ {1 / 2}}\right). \tag {21}
$$
A.3.2 Discussion on the bias bound
More direct upper bound. The precedent derivations can be made more direct with the following series of implications, with  $A \preceq B$  meaning that  $x^{\top}Ax \leq x^{\top}Bx$  for every  $x$ :
$$
\begin{array}{l} \left\| F _ {n} \right\| \leq 1 / 2 \quad \Rightarrow \quad - 1 / 2 I \preceq F _ {n} \preceq 1 / 2 I \\ \Rightarrow \quad 1 / 2 I \preceq I - F _ {n} \preceq 3 / 2 I \\ \Rightarrow \quad 4 / 9 I \preceq (I - F _ {n}) ^ {- 2} \preceq 4 I \\ \Rightarrow \quad 4 / 9 \mathcal {S} (K) \preceq \theta_ {*} ^ {\top} \Sigma^ {1 / 2} \Sigma_ {1} ^ {- 1} \left(I - F _ {n}\right) ^ {- 2} \Sigma_ {1} ^ {- 1} \Sigma^ {1 / 2} \theta_ {*} \preceq 4 \mathcal {S} (K). \\ \end{array}
$$
Let us assume that
$$
\begin{array}{l} \mathbb {E} \left[ (I - F _ {n}) ^ {- 1} \Sigma \Sigma_ {1} ^ {- 1} (I - F _ {n}) ^ {- 1} \mid \| F _ {n} \| \leq 1 / 2 \right] \\ \preceq c ^ {2} \mathbb {E} \left[ \Sigma^ {1 / 2} \Sigma_ {1} ^ {- 1 / 2} \left(I - F _ {n}\right) ^ {- 2} \Sigma^ {1 / 2} \Sigma_ {1} ^ {- 1 / 2} \mid \| F _ {n} \| \leq 1 / 2 \right], \tag {22} \\ \end{array}
$$
This leads to the simple upper bound
$$
\begin{array}{l} \mathcal {B} _ {n} = \theta_ {*} ^ {\top} \Sigma_ {1} ^ {- 1 / 2} (I - F _ {n}) ^ {- 1} \Sigma_ {1} ^ {- 1} \Sigma (I - F _ {n}) ^ {- 1} \Sigma_ {1} ^ {- 1 / 2} \theta_ {*} \\ \leq c ^ {2} \theta_ {*} ^ {\top} \Sigma_ {1} ^ {- 1 / 2} (\Sigma_ {1} ^ {- 1}) ^ {1 / 2} (I - F _ {n}) ^ {- 2} (\Sigma_ {1} ^ {- 1}) ^ {1 / 2} \Sigma_ {1} ^ {- 1 / 2} \theta_ {*} + \| f ^ {*} \| _ {L ^ {2}} ^ {2} \mathbb {P} (\| F _ {n} \| > 1 / 2) \\ \leq 4 c ^ {2} \mathcal {S} (K) + \| f ^ {*} \| _ {2} ^ {2} \mathcal {N} _ {1} (K) \exp \left(- \frac {n}{6 \mathcal {N} _ {+} (K)}\right). \\ \end{array}
$$
Improvement directions. In essence, we expect the bias upper-lower bound to behave as
$$
\mathcal {S} (K) (I - F _ {n}) ^ {- 2} - I \simeq \mathcal {S} (K) F _ {n}.
$$
Getting this linear dependency in  $F_{n}$  explicitly would allow to improve the bound since
$$
\mathbb {E} _ {\mathcal {D} _ {n}} \left[ \left\| F _ {n} \right\| \mid \left\| F _ {n} \right\| \leq 1 / 2 \right] \lesssim \mathcal {N} (K) \mathcal {N} _ {+} (K) n ^ {- 1}.
$$
Moreover, going back to the definition of  $F_{n}$
$$
\begin{array}{l} \mathbb {E} _ {\mathcal {D} _ {n}} \theta_ {*} ^ {\top} \Sigma \Sigma_ {1} ^ {- 1} F _ {n} \Sigma \Sigma_ {1} ^ {- 1} \theta_ {*} = \mathbb {E} _ {\mathcal {D} _ {n}} \left[ \left(\frac {1}{n} \sum_ {i = 1} ^ {n} \theta_ {*} ^ {\top} \Sigma \Sigma_ {1} ^ {- 3 / 2} \varphi (X _ {i})\right) ^ {2} \right] - \mathbb {E} _ {X} \left[ \theta_ {*} ^ {\top} \Sigma \Sigma_ {1} ^ {- 3 / 2} \varphi (X) \right] ^ {2} \\ = \mathbb {E} _ {\mathcal {D} _ {n}} \left[ \left(\frac {1}{n} \sum_ {i = 1} ^ {n} \theta_ {*} ^ {\top} \Sigma \Sigma_ {1} ^ {- 3 / 2} \varphi (X _ {i}) - \mathbb {E} _ {X} \left[ \theta_ {*} ^ {\top} \Sigma \Sigma_ {1} ^ {- 3 / 2} \varphi (X) \right]\right) ^ {2} \right], \\ \end{array}
$$
which suggests possible improvements of the bound in  $\mathcal{S}(K)n^{-1}$ .
A.3.3 Bounding the variance term
Let us now work on the term  $\mathcal{V}_n = \mathrm{Tr}\Sigma (\Sigma_n + 1)^{-2}\Sigma_n$ . It works similarly to the bias term, concentrating towards  $\mathcal{N}_2(K)$ . With  $\Sigma_{n,1} = \Sigma_{n} + 1$ , we have
$$
\mathrm {T r} \left(\Sigma \Sigma_ {n} \Sigma_ {n, 1} ^ {- 2} - \Sigma^ {2} \Sigma_ {1} ^ {- 2}\right) = \mathrm {T r} \left(\Sigma \Sigma_ {1} ^ {- 1} \Sigma_ {n} \Sigma_ {n, 1} ^ {- 1} (\Sigma_ {n, 1} ^ {- 1} \Sigma_ {1} - I) + \Sigma \Sigma_ {1} ^ {- 1} (\Sigma_ {n} \Sigma_ {n, 1} ^ {- 1} - \Sigma \Sigma_ {1} ^ {- 1})\right).
$$
Using that, for any  $A$  positive semi-definite and any  $B$ ,  $\operatorname{Tr}(AB) \leq \| B \| \operatorname{Tr}(A)$ , it follows that
$$
\begin{array}{l} \left| \mathcal {V} _ {n} - \mathcal {N} _ {2} (K) \right| \leq \mathcal {N} _ {1} (K) \left\| \Sigma_ {n} \Sigma_ {n, 1} ^ {- 1} \right\| \left\| \Sigma_ {n, 1} ^ {- 1} \Sigma_ {1} - I \right\| + \mathcal {N} _ {1} (K) \left\| \Sigma_ {n} \Sigma_ {n, 1} ^ {- 1} - \Sigma \Sigma_ {1} ^ {- 1} \right\| \\ \leq \mathcal {N} _ {1} (K) \left\| \Sigma_ {n, 1} ^ {- 1} \Sigma_ {1} - I \right\| + \mathcal {N} _ {1} (K) \left\| \Sigma_ {n} \Sigma_ {n, 1} ^ {- 1} - \Sigma \Sigma_ {1} ^ {- 1} \right\|. \\ \end{array}
$$
Let us focus on the first term. Using  $a^{-1} - b^{-1} = a^{-1}(b - a)b^{-1}$ , we get
$$
\begin{array}{l} \left| \left| \Sigma_ {1} \Sigma_ {n, 1} ^ {- 1} - I \right| \right| = \left| \left| \Sigma_ {1} \Sigma_ {n, 1} ^ {- 1} (\Sigma - \Sigma_ {n}) \Sigma_ {1} ^ {- 1} \right| \right| \\ \leq \left\| \Sigma_ {1} \Sigma_ {n, 1} ^ {- 1} \right\| \left\| \Sigma_ {1} ^ {1 / 2} F _ {n} \Sigma_ {1} ^ {- 1 / 2} \right\| \\ \leq \left(\| I \| + \left\| \Sigma_ {1} \Sigma_ {n, 1} ^ {- 1} - I \right\|\right) \left\| \Sigma_ {1} ^ {1 / 2} F _ {n} \Sigma_ {1} ^ {- 1 / 2} \right\| \\ \leq \sum_ {i > 0} \left\| \Sigma_ {1} ^ {1 / 2} F _ {n} \Sigma_ {1} ^ {- 1 / 2} \right\| ^ {i}. \\ \end{array}
$$
$$
\begin{array}{l} \left\| \Sigma_ {n} \Sigma_ {n, 1} ^ {- 1} - \Sigma \Sigma_ {1} ^ {- 1} \right\| \leq \left\| \Sigma_ {n} \left(\Sigma_ {n, 1} ^ {- 1} - \Sigma_ {1} ^ {- 1} \right. \right\| + \left\| (\Sigma_ {n} - \Sigma) \Sigma_ {1} ^ {- 1} \right\| \\ \leq \left\| \Sigma_ {n} \Sigma_ {n, 1} ^ {- 1} (\Sigma - \Sigma_ {n}) \Sigma_ {1} ^ {- 1} \right\| + \left\| (\Sigma_ {n} - \Sigma) \Sigma_ {1} ^ {- 1} \right\| \\ \leq 2 \left| \left| \left(\Sigma_ {n} - \Sigma\right) \Sigma_ {1} ^ {- 1} \right| \right| \\ = 2 \left\| \Sigma_ {1} ^ {1 / 2} F _ {n} \Sigma_ {1} ^ {- 1 / 2} \right\|. \\ \end{array}
$$
Similarly as for (19), if we assume that
$$
\left\| \Sigma_ {1} ^ {1 / 2} F _ {n} \Sigma_ {1} ^ {- 1 / 2} \right\| \leq c \| F _ {n} \|, \tag {23}
$$
then we have, as long as  $\| F_n\| \leq 1 / 2c$
$$
\begin{array}{l} | \mathcal {V} _ {n} - \mathcal {N} _ {2} (K) | \leq \sum_ {i > 0} \left\| \Sigma_ {1} ^ {1 / 2} F _ {n} \Sigma_ {1} ^ {- 1 / 2} \right\| ^ {i} + 2 \left\| \Sigma_ {1} ^ {1 / 2} F _ {n} \Sigma_ {1} ^ {- 1 / 2} \right\| \\ \leq \frac {c \| F _ {n} \|}{1 - c \| F _ {n} \|} + 2 c \| F _ {n} \| \leq 4 c \| F _ {n} \|. \\ \end{array}
$$
For the case when  $\| F_n\|$  is large, we can again proceed with a simple bound:
$$
\left| \mathcal {V} _ {n} - \mathcal {N} _ {2} (K) \right| \leq \left| \operatorname {T r} \left(\Sigma \Sigma_ {n} (\Sigma_ {n} + 1) ^ {- 2}\right) \right| + \operatorname {T r} \left(\Sigma^ {2} \Sigma_ {1} ^ {- 2}\right) \leq 2 \operatorname {T r} (\Sigma).
$$
Splitting the full expectation as we did for the bias we thus obtain