SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathbb {E} _ {D} R _ {P} \left(f _ {t, \rho}\right) - R _ {P} ^ {*} \geq \frac {\sigma^ {2}}{n} \sum_ {i = 1} ^ {n} \mathbb {E} _ {\boldsymbol {X}} \frac {\lambda_ {i} \left(k _ {*} (\boldsymbol {X} , \boldsymbol {X}) / n\right) \left(1 - e ^ {- 2 t \left(\lambda_ {i} (k (\boldsymbol {X} , \boldsymbol {X}) / n) + \rho\right)}\right) ^ {2}}{\left(\lambda_ {i} (k (\boldsymbol {X} , \boldsymbol {X}) / n) + \rho\right) ^ {2}}. \tag {3}
$$
Proof. Recall from Eq. (1) that
$$
f _ {t, \rho} (\boldsymbol {x}) = k (\boldsymbol {x}, \boldsymbol {X}) \boldsymbol {A} _ {t, \rho} (\boldsymbol {X}) \boldsymbol {y},
$$
$$
\boldsymbol {A} _ {t, \rho} (\boldsymbol {X}) := \left(\boldsymbol {I} _ {n} - e ^ {- \frac {2}{n} t (k (\boldsymbol {X}, \boldsymbol {X}) + \rho n \boldsymbol {I} _ {n})}\right) \left(k (\boldsymbol {X}, \boldsymbol {X}) + \rho n \boldsymbol {I} _ {n}\right) ^ {- 1}.
$$
By setting  $(\pmb{v}_{\pmb{X},\pmb{x}})^{\top} \coloneqq k(\pmb{x},\pmb{X})\pmb{A}_{t,\rho}(\pmb{X})$ , we can write  $f_{\pmb{X},\pmb{y},t,\rho}(\pmb{x}) \coloneqq f_{t,\rho}(\pmb{x}) = (\pmb{v}_{\pmb{X},\pmb{x}})^{\top}\pmb{y}$ . Using Lemma F.1, we then obtain
$$
\begin{array}{l} \mathbb {E} _ {D} R _ {P} \left(f _ {\boldsymbol {X}, \boldsymbol {y}, t, \rho}\right) - R _ {P} ^ {*} \geq \sigma^ {2} \mathbb {E} _ {\boldsymbol {X}, \boldsymbol {x}} \operatorname {t r} \left(\boldsymbol {v} _ {\boldsymbol {X}, \boldsymbol {x}} \left(\boldsymbol {v} _ {\boldsymbol {X}, \boldsymbol {x}}\right) ^ {\top}\right) \\ = \sigma^ {2} \mathbb {E} _ {\boldsymbol {X}, \boldsymbol {x}} \operatorname {t r} \left(\boldsymbol {A} _ {t, \rho} (\boldsymbol {X}) ^ {\top} k (\boldsymbol {X}, \boldsymbol {x}) k (\boldsymbol {x}, \boldsymbol {X}) \boldsymbol {A} _ {t, \rho} (\boldsymbol {X})\right). \\ \end{array}
$$
$$
\left. \left(\mathbb {E} _ {\boldsymbol {x}} k (\boldsymbol {X}, \boldsymbol {x}) k (\boldsymbol {x}, \boldsymbol {X})\right) _ {i j} = \mathbb {E} _ {\boldsymbol {x}} k \left(\boldsymbol {x} _ {i}, \boldsymbol {x}\right) k \left(\boldsymbol {x}, \boldsymbol {x} _ {j}\right) = k _ {*} \left(\boldsymbol {x} _ {i}, \boldsymbol {x} _ {j}\right) = k _ {*} \left(\boldsymbol {X}, \boldsymbol {X}\right) _ {i j}, \right.
$$
we conclude
$$
\begin{array}{l} \mathbb {E} _ {D} R _ {P} \left(f _ {\boldsymbol {X}, \boldsymbol {y}, t, \rho}\right) - R _ {P} ^ {*} \geq \sigma^ {2} \mathbb {E} _ {\boldsymbol {X}} \operatorname {t r} \left(\boldsymbol {A} _ {t, \rho} ^ {\top} k _ {*} (\boldsymbol {X}, \boldsymbol {X}) \boldsymbol {A} _ {t, \rho}\right) \\ = \sigma^ {2} \mathbb {E} _ {\boldsymbol {X}} \operatorname {t r} \left(k _ {*} (\boldsymbol {X}, \boldsymbol {X}) \boldsymbol {A} _ {t, \rho} (\boldsymbol {X}) \boldsymbol {A} _ {t, \rho} (\boldsymbol {X}) ^ {\top}\right). \\ \end{array}
$$
Richter (1958) showed (see also Mirsky, 1959) that for two symmetric matrices  $B, C$ , we have  $\operatorname{tr}(BC) \geq \sum_{i=1}^{n} \lambda_i(B) \lambda_{n+1-i}(C)$ . We can therefore conclude
$$
\mathbb {E} _ {D} R _ {P} \left(f _ {\boldsymbol {X}, \boldsymbol {y}, t, \rho}\right) - R _ {P} ^ {*} \geq \sigma^ {2} \mathbb {E} _ {\boldsymbol {X}} \sum_ {i = 1} ^ {n} \lambda_ {i} \left(k _ {*} (\boldsymbol {X}, \boldsymbol {X})\right) \lambda_ {n + 1 - i} \left(\boldsymbol {A} _ {t, \rho} (\boldsymbol {X}) \boldsymbol {A} _ {t, \rho} (\boldsymbol {X}) ^ {\top}\right).
$$
As  $\mathbf{A}_{t,\rho}(\mathbf{X})\mathbf{A}_{t,\rho}(\mathbf{X})^\top$  is built only out of the matrices  $k(\mathbf{X},\mathbf{X})$  and  $I_{n}$ , it is not hard to see that  $\mathbf{A}_{t,\rho}(\mathbf{X})\mathbf{A}_{t,\rho}(\mathbf{X})^\top$  has the same eigenbasis as  $k(\mathbf{X},\mathbf{X})$  with eigenvalues
$$
\tilde {\lambda} _ {i} := \left(\frac {1 - e ^ {- \frac {2}{n} t (\lambda_ {i} (k (\boldsymbol {X} , \boldsymbol {X})) + \rho n)}}{\lambda_ {i} (k (\boldsymbol {X} , \boldsymbol {X})) + \rho n}\right) ^ {2} = \frac {1}{n ^ {2}} \left(\frac {1 - e ^ {- 2 t (\lambda_ {i} (k (\boldsymbol {X} , \boldsymbol {X}) / n) + \rho)}}{\lambda_ {i} (k (\boldsymbol {X} , \boldsymbol {X}) / n) + \rho}\right) ^ {2}.
$$
It remains to order these eigenvalues correctly. To this end, we observe that for  $\lambda > 0$ , the function  $g(\lambda) \coloneqq \frac{1 - e^{-2t\lambda}}{\lambda}$  satisfies
$$
g ^ {\prime} (\lambda) = \frac {2 t \lambda e ^ {- 2 t \lambda} - (1 - e ^ {- 2 t \lambda})}{\lambda^ {2}} = \frac {(2 t \lambda + 1) e ^ {- 2 t \lambda} - 1}{\lambda^ {2}} \leq \frac {e ^ {2 t \lambda} e ^ {- 2 t \lambda} - 1}{\lambda^ {2}} = 0.
$$
Therefore,  $g$  is nonincreasing, hence the sequence  $(\tilde{\lambda}_i)$  is nondecreasing and thus
$$
\lambda_ {n + 1 - i} (\pmb {A} _ {t, \rho} \pmb {A} _ {t, \rho} ^ {\top}) = \tilde {\lambda} _ {i},
$$
from which the claim follows.
Theorem F.2. Let  $k$  be a kernel on a compact set  $\Omega$  and let  $P_X$  be supported on  $\Omega$ . Suppose that  $k(\mathbf{X},\mathbf{X})$  is almost surely positive definite and that  $\operatorname{Var}(y|\mathbf{x}) \geq \sigma^2$  for  $P_X$ -almost all  $\mathbf{x}$ . Fix constants  $c > 0$  and  $q, C \geq 1$ . Suppose that  $\lambda_i \coloneqq \lambda_i(T_{k,P_X}) \geq ci^{-q}$ . Let  $\mathcal{I}(n)$  be the set of all  $i \in [n]$  for which
$$
\lambda_ {i} / C \leq \lambda_ {i} (k (\boldsymbol {X}, \boldsymbol {X}) / n) \leq C \lambda_ {i} \tag {F.1}
$$
$$
\lambda_ {i} ^ {2} / C \leq \lambda_ {i} \left(k _ {*} (\boldsymbol {X}, \boldsymbol {X}) / n\right)
$$
both hold at the same time with probability  $\geq 1/2$ . Moreover, let  $I(n) \coloneqq \max \{m \in [n] \mid [m] \subseteq \mathcal{I}(n)\}$ . Then, there exists a constant  $c' > 0$  depending only on  $c, C$  such that for all  $\rho \in [0, \infty)$  and  $t \in (0, \infty]$ , the following two bounds hold:
$$
\mathbb {E} _ {D} R _ {P} \left(f _ {\boldsymbol {X}, \boldsymbol {y}, t, \rho}\right) - R _ {P} ^ {*} \geq c ^ {\prime} \sigma^ {2} \frac {1}{1 + (\rho + t ^ {- 1}) n ^ {q}} \cdot \frac {| \mathcal {I} (n) |}{n},
$$
$$
\mathbb {E} _ {D} R _ {P} (f _ {\boldsymbol {X}, \boldsymbol {y}, t, \rho}) - R _ {P} ^ {*} \geq c ^ {\prime} \sigma^ {2} \min  \left\{\frac {(\rho + t ^ {- 1}) ^ {- 2}}{n}, \frac {(\rho + t ^ {- 1}) ^ {- 1 / q}}{n}, \frac {I (n)}{n} \right\}.
$$
Remark F.3. Theorem F.2 provides two lower bounds, one for general "concentration sets"  $\mathcal{I}(n)$  and one that applies if concentration holds for a sequence of "head eigenvalues"  $\{1,\dots,I(n)\} \subseteq \mathcal{I}(n)$ . If  $I(n)\approx |\mathcal{I}(n)|$ , the latter bound is stronger for larger regularization levels, and this bound would be particularly suitable for typical forms of relative concentration inequalities for kernel matrices. However, in this paper, we obtain concentration only for "middle eigenvalues"  $\mathcal{I}(n) = \{i\mid \varepsilon n\leq i\leq (1 - \varepsilon)n\}$ , and therefore we only use the first bound in the proof of Theorem 6.
Proof of Theorem F.2. Step 1: Miscellaneous inequalities. For  $x > 0$
$$
1 - e ^ {- x} = 1 - \frac {1}{e ^ {x}} \geq 1 - \frac {1}{x + 1} = \frac {x}{x + 1} = \frac {1}{1 + x ^ {- 1}}. \tag {F.2}
$$
Moreover, since  $(1 + a)^2 \leq (1 + a)^2 + (1 - a)^2 = 2 + 2a^2$ , we have for  $a \neq -1$ :
$$
\left(\frac {1}{1 + a}\right) ^ {2} \geq \frac {1}{2 \left(1 + a ^ {2}\right)}. \tag {F.3}
$$
Step 2: Applying the eigenvalue bound. Define
$$
S _ {i} (\pmb {X}) := \frac {\lambda_ {i} (k _ {*} (\pmb {X} , \pmb {X}) / n) \left(1 - e ^ {- 2 t (\lambda_ {i} (k (\pmb {X} , \pmb {X}) / n) + \rho)}\right) ^ {2}}{(\lambda_ {i} (k (\pmb {X} , \pmb {X}) / n) + \rho) ^ {2}}.
$$
By Proposition 5, we have
$$
\mathbb {E} _ {D} R _ {P} \left(f _ {\boldsymbol {X}, \boldsymbol {y}, t, \rho}\right) - R _ {P} ^ {*} \geq \frac {\sigma^ {2}}{n} \sum_ {i = 1} ^ {n} \mathbb {E} _ {\boldsymbol {X}} S _ {i} (\boldsymbol {X}) \geq \frac {\sigma^ {2}}{n} \sum_ {i \in \mathcal {I} (n)} \mathbb {E} _ {\boldsymbol {X}} S _ {i} (\boldsymbol {X}). \tag {F.4}
$$
Since  $S_{i}(\mathbf{X})$  is almost surely positive, we can focus on the case where (F.1) and (F.2) hold, which is true with probability  $\geq 1/2$  by assumption for  $i \in \mathcal{I}(n)$ . Hence,
$$
\mathbb {E} _ {\boldsymbol {X}} S _ {i} (\boldsymbol {X}) \geq \frac {1}{2} \frac {\lambda_ {i} ^ {2} / C \cdot (1 - e ^ {- 2 t (\lambda_ {i} / C + \rho)}) ^ {2}}{(C \lambda_ {i} + \rho) ^ {2}} \stackrel {\text {(F . 2)}} {\geq} \frac {1}{2} \frac {\lambda_ {i} ^ {2} / C}{(C \lambda_ {i} + \rho) ^ {2} (1 + (2 t (\lambda_ {i} / C + \rho)) ^ {- 1}) ^ {2}}.
$$
We can upper-bound the denominator, using  $C \geq 1$ , as
$$
\begin{array}{l} \left(C \lambda_ {i} + \rho\right) \left(1 + \frac {1}{2 t \left(\lambda_ {i} / C + \rho\right)}\right) \leq C \lambda_ {i} + \rho + \frac {C ^ {2} \lambda_ {i} + C \rho}{\left(\lambda_ {i} + C \rho\right) t} \leq C \lambda_ {i} + \rho + \frac {C ^ {2} \lambda_ {i} + C ^ {3} \rho}{\left(\lambda_ {i} + C \rho\right) t} \\ \leq C ^ {2} \left(\lambda_ {i} + \rho + t ^ {- 1}\right), \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {E} _ {\boldsymbol {X}} S _ {i} (\boldsymbol {X}) \geq \frac {1}{2 C ^ {5}} \frac {\lambda_ {i} ^ {2}}{(\lambda_ {i} + \rho + t ^ {- 1}) ^ {2}} = \frac {1}{2 C ^ {5}} \frac {1}{\left(1 + \frac {\rho + t ^ {- 1}}{\lambda_ {i}}\right) ^ {2}} \stackrel {(F. 3)} {\geq} \frac {1}{4 C ^ {5}} \frac {1}{1 + \left(\frac {\rho + t ^ {- 1}}{\lambda_ {i}}\right) ^ {2}} \\ \geq \frac {1}{4 C ^ {5}} \frac {1}{1 + \left(\frac {\rho + t ^ {- 1}}{c} i ^ {q}\right) ^ {2}}. \tag {F.5} \\ \end{array}
$$
Step 3: Analyzing the sum. We want to analyze the behavior of the sum
$$
S (\beta) := \sum_ {i \in \mathcal {I} (n)} \frac {1}{1 + (\beta i ^ {q}) ^ {2}}
$$
for  $\beta \coloneqq \frac{\rho + t^{-1}}{c} > 0$ . We first obtain the trivial bound
$$
S (\beta) \geq | \mathcal {I} (n) | \frac {1}{1 + (\beta n ^ {q}) ^ {2}}.
$$
Moreover, we can bound
$$
S (\beta) \geq \sum_ {i = 1} ^ {I (n)} \frac {1}{1 + (\beta i ^ {q}) ^ {2}}
$$
and distinguish three cases:
(a) If  $\beta \geq 1$ , we bound
$$
S (\beta) \geq \sum_ {i = 1} ^ {I (n)} \frac {1}{2 \left(\beta i ^ {q}\right) ^ {2}} \geq \frac {1}{2 \beta^ {2}}.
$$
(b) If  $\beta \in (I(n)^{-q},1)$ , we observe that
$$
J (\beta) := \lfloor \beta^ {- 1 / q} \rfloor \geq \lceil \beta^ {- 1 / q} \rceil - 1 \geq \frac {1}{2} \lceil \beta^ {- 1 / q} \rceil \geq \frac {\beta^ {- 1 / q}}{2}
$$
$$
S (\beta) \geq \sum_ {i = 1} ^ {J (\beta)} \frac {1}{1 + (\beta i ^ {q}) ^ {2}} \geq \sum_ {i = 1} ^ {J (\beta)} \frac {1}{1 + 1} = \frac {J (\beta)}{2} \geq \frac {\beta^ {- 1 / q}}{4}.
$$
(c) If  $\beta \in (0, I(n)^{-q}]$ , we similarly find that
$$
S (\beta) \geq \sum_ {i = 1} ^ {I (n)} \frac {1}{1 + 1} = \frac {I (n)}{2}.
$$
Moreover, there is an absolute constant  $c_{1} > 0$  such that for any  $\beta >0$
$$
S (\beta) \geq c _ {1} \min  \left\{\beta^ {- 2}, \beta^ {- 1 / q}, I (n) \right\}, \tag {F.6}
$$
(a)  $\beta^{-2} = \min \{\beta^{-2},\beta^{-1 / q},I(n)\}$  for  $\beta \geq 1$
(b)  $\beta^{-1 / q} = \min \{\beta^{-2},\beta^{-1 / q},I(n)\}$  for  $\beta \in (I(n)^{-q},1)$ , and
(c)  $I(n) = \min \{\beta^{-2},\beta^{-1 / q},I(n)\}$  for  $\beta \in (0,I(n)^{-q}]$
Step 4: Putting it together. Combining the trivial bound in Step 3 with Eq. (F.4) and Eq. (F.5), we obtain
$$
\begin{array}{l} \mathbb {E} _ {D} R _ {P} (f _ {\boldsymbol {X}, \boldsymbol {y}, t, \rho}) - R _ {P} ^ {*} \geq \frac {\sigma^ {2}}{n} \sum_ {i \in \mathcal {I} (n)} \mathbb {E} _ {\boldsymbol {X}} S _ {i} (\boldsymbol {X}) \geq \frac {\sigma^ {2}}{n} \cdot \frac {1}{4 C ^ {5}} S (\beta) \\ \geq c ^ {\prime} \sigma^ {2} \frac {1}{1 + (\rho + t ^ {- 1}) n ^ {q}} \cdot \frac {| \mathcal {I} (n) |}{n} \tag {F.7} \\ \end{array}
$$
for a suitable constant  $c' > 0$  depending only on  $c$  and  $C$ .
Moreover, from Eq. (F.6), we obtain
$$
S (\beta) \geq \tilde {c} _ {1} \min  \left\{\beta^ {- 2}, \beta^ {- 1 / q}, I (n) \right\} \geq \tilde {c} ^ {\prime \prime} \min  \left\{\left(\rho + t ^ {- 1}\right) ^ {- 2}, \left(\rho + t ^ {- 1}\right) ^ {- 1 / q}, I (n) \right\}
$$
for a suitable constant  $\tilde{c}'' > 0$  depending only on  $c$ . Again, (F.4) and (F.5) yield
$$
\begin{array}{l} \mathbb {E} _ {D} R _ {P} \left(f _ {\boldsymbol {X}, \boldsymbol {y}, t, \rho}\right) - R _ {P} ^ {*} \geq \frac {\sigma^ {2}}{n} \cdot \frac {1}{4 C ^ {5}} S (\beta) \\ \geq \frac {\tilde {c} ^ {\prime \prime}}{4 C ^ {5}} \sigma^ {2} \min \left\{\frac {(\rho + t ^ {- 1}) ^ {- 2}}{n}, \frac {(\rho + t ^ {- 1}) ^ {- 1 / q}}{n}, \frac {I (n)}{n} \right\}. \\ \end{array}
$$
F.2 Equivalences of norms and eigenvalues
Later, we will use concentration inequalities for kernel matrix eigenvalues proved for specific kernels, which we then want to transfer to other kernels with equivalent RKHSs. In this subsection, we show that this is possible.
Definition F.4 ( $C$ -equivalence of matrices and norms). Let  $n \geq 1$  and let  $\pmb{K}, \tilde{\pmb{K}} \in \mathbb{R}^{n \times n}$  be symmetric. For  $C \geq 1$ , we say that  $\pmb{K}$  and  $\tilde{\pmb{K}}$  are  $C$ -equivalent if their ordered eigenvalues satisfy
$$
C ^ {- 1} \lambda_ {i} (\boldsymbol {K}) \leq \lambda_ {i} (\tilde {\boldsymbol {K}}) \leq C \lambda_ {i} (\boldsymbol {K})
$$
for all  $i \in [n]$ . Moreover, we say that two norms  $\| \cdot \|_A$ ,  $\| \cdot \|_B$  on a vector space  $V$  are  $C$ -equivalent if
$$
C ^ {- 1} \| \boldsymbol {v} \| _ {A} \leq \| \boldsymbol {v} \| _ {B} \leq C \| \boldsymbol {v} \| _ {A}
$$
for all  $\pmb{v} \in V$ .
Lemma F.5. Let  $n \geq 1$  and let  $K, \tilde{K} \in \mathbb{R}^{n \times n}$  be symmetric. Then,  $K$  and  $\tilde{K}$  are  $C$ -equivalent iff the Moore-Penrose pseudoinverses  $K^+$  and  $\tilde{K}^+$  are  $C$ -equivalent.
Proof. This follows from the fact that if  $\pmb{K}$  has eigenvalues  $\lambda_1, \dots, \lambda_n$ , then  $K^+$  has eigenvalues  $1 / \lambda_1, \dots, 1 / \lambda_n$ , where we define  $1 / 0 \coloneqq 0$ . (A detailed proof would be a bit technical due to the sorting of eigenvalues.)
Lemma F.6. Let  $k:\mathcal{X}\times \mathcal{X}\to \mathbb{R}$  be a kernel on a set  $\mathcal{X}$ . Then, for any  $\pmb {y}\in \mathbb{R}^n$
$$
\boldsymbol {y} ^ {\top} k (\boldsymbol {X}, \boldsymbol {X}) ^ {+} \boldsymbol {y} = \| f _ {k, \boldsymbol {y}} ^ {*} \| _ {\mathcal {H} _ {k}} ^ {2},
$$
where  $\mathcal{H}_k$  is the RKHS associated with  $k$  and  $f_{k,y}^{*}$  is the minimum-norm regression solution
$$
f_{k,\boldsymbol{y}}^{*}:= \operatorname *{argmin}_{f\in B}\| f\|_{\mathcal{H}_{k}}^{2},
$$
$$
B := \left\{f \in \mathcal {H} _ {k} \mid \sum_ {i = 1} ^ {n} (f (\boldsymbol {x} _ {i}) - y _ {i}) ^ {2} = \inf  _ {\tilde {f} \in \mathcal {H} _ {k}} \sum_ {i = 1} ^ {n} (\tilde {f} (\boldsymbol {x} _ {i}) - y _ {i}) ^ {2} \right\}.
$$
Proof. It is well-known that  $f_{k,\pmb{y}}^{*}(\pmb{x}) = \sum_{i=1}^{n} \alpha_{i} k(\pmb{x},\pmb{x}_{i})$ , where  $\pmb{\alpha} \coloneqq \pmb{K}^{+}\pmb{y}$  (see e.g. Rangamani et al., 2023). We then have
$$
\begin{array}{l} \left\| f _ {k, \boldsymbol {y}} ^ {*} \right\| _ {\mathcal {H} _ {k}} ^ {2} = \left\langle \sum_ {i = 1} ^ {n} \alpha_ {i} k (\boldsymbol {x} _ {i}, \cdot), \sum_ {j = 1} ^ {n} \alpha_ {j} k (\boldsymbol {x} _ {j}, \cdot) \right\rangle_ {\mathcal {H} _ {k}} = \sum_ {i = 1} ^ {n} \sum_ {j = 1} ^ {n} \alpha_ {i} \boldsymbol {K} _ {i j} \alpha_ {j} \\ = \boldsymbol {y} ^ {\top} \boldsymbol {K} ^ {+} \boldsymbol {K} \boldsymbol {K} ^ {+} \boldsymbol {y} = \boldsymbol {y} ^ {\top} \boldsymbol {K} ^ {+} \boldsymbol {y}, \\ \end{array}
$$
where the last step follows from a standard identity for the Moore-Penrose pseudoinverse (see e.g. Section 1.1.1 in Wang et al., 2018).
Lemma F.7. Let  $\mathcal{H}_1$  and  $\mathcal{H}_2$  be two RKHSs with  $\mathcal{H}_1 \subset \mathcal{H}_2$ . Then there exists a constant  $C > 0$  such that  $\|f\|_{\mathcal{H}_2} \leq C \|f\|_{\mathcal{H}_1}$ .
Proof. Let  $I: \mathcal{H}_1 \to \mathcal{H}_2$  be the inclusion map, i.e.  $I_h := h$  for all  $h \in \mathcal{H}_1$ . Obviously,  $I$  is linear and we need to show that  $I$  is bounded. To this end, let  $(h_n)_{n \geq 1} \subset \mathcal{H}_1$  be a sequence such that there exist  $h \in \mathcal{H}_1$  and  $g \in \mathcal{H}_2$  with  $h_n \to h$  in  $\mathcal{H}_1$  and  $Ih_n \to g$  in  $\mathcal{H}_2$ . This implies  $h_n \to h$  pointwise and  $h_n = Ih_n \to g$  pointwise, which in turn gives  $h = g$ . The closed graph theorem, see e.g. (Megginson, 1998, Theorem 1.6.11), then shows that  $I$  is bounded.
Applying Lemma F.7 twice shows that RKHSs  $\mathcal{H}_1$  and  $\mathcal{H}_2$  with  $\mathcal{H}_1 = \mathcal{H}_2$  automatically have  $C$ -equivalent norms for a suitable constant  $C \geq 1$ . The following result investigates the corresponding kernels.
Proposition F.8 (Equivalent kernels have equivalent kernel matrices). Let  $k, \tilde{k} : \mathcal{X} \times \mathcal{X} \to \mathbb{R}$  be kernels such that their RKHSs are equal as sets and the corresponding RKHS-norms are  $C$ -equivalent as defined in Definition F.4. Then, for any  $n \geq 1$  and any  $x_1, \ldots, x_n \in \mathcal{X}$ , the corresponding kernel matrices  $k(\boldsymbol{X}, \boldsymbol{X}), \tilde{k}(\boldsymbol{X}, \boldsymbol{X})$  are  $C^2$ -equivalent.
Proof. Let  $i \in [n]$ . For  $\pmb{y} \in \mathbb{R}^n$  we have, using the notation of Lemma F.6:
$$
\boldsymbol {y} ^ {\top} k (\boldsymbol {X}, \boldsymbol {X}) ^ {+} \boldsymbol {y} = \| f _ {k, \boldsymbol {y}} ^ {*} \| _ {\mathcal {H} _ {k}} ^ {2} \geq C ^ {- 2} \| f _ {k, \boldsymbol {y}} ^ {*} \| _ {\mathcal {H} _ {\tilde {k}}} ^ {2} \geq C ^ {- 2} \| f _ {\tilde {k}, \boldsymbol {y}} ^ {*} \| _ {\mathcal {H} _ {\tilde {k}}} ^ {2} = C ^ {- 2} \boldsymbol {y} ^ {\top} \tilde {k} (\boldsymbol {X}, \boldsymbol {X}) ^ {+} \boldsymbol {y}.
$$
Now, by the Courant-Fischer-Weyl theorem,
$$
\begin{array}{l} \lambda_ {i} (k (\boldsymbol {X}, \boldsymbol {X}) ^ {+}) = \sup  _ {V: \dim V = i} \inf  _ {y \in V: \| y \| _ {2} = 1} y ^ {\top} k (\boldsymbol {X}, \boldsymbol {X}) ^ {+} y \\ \geq C ^ {- 2} \sup  _ {V: \dim V = i} \inf  _ {y \in V: \| y \| _ {2} = 1} y ^ {\top} \tilde {k} (\boldsymbol {X}, \boldsymbol {X}) ^ {+} y \\ = C ^ {- 2} \lambda_ {i} (\tilde {k} (\boldsymbol {X}, \boldsymbol {X}) ^ {+}). \\ \end{array}
$$
By switching the roles of  $k$  and  $\tilde{k}$ , we obtain that  $k(\mathbf{X},\mathbf{X})^{+}$  and  $\tilde{k} (\mathbf{X},\mathbf{X})^{+}$  are  $C^2$ -equivalent. By Lemma F.5  $k(\mathbf{X},\mathbf{X})$  and  $\tilde{k} (\mathbf{X},\mathbf{X})$  are then also  $C^2$ -equivalent.
To prove Theorem 6 for arbitrary input distributions  $P_{X}$  with lower and upper bounded densities, we need the following theorem investigating the corresponding eigenvalues of the integral operator.
Lemma F.9 (Integral operators for equivalent densities have equivalent eigenvalues). Let  $k: \mathcal{X} \times \mathcal{X} \to \mathbb{R}$  be a kernel and let  $\mu, \nu$  be finite measures on  $\mathcal{X}$  whose support is  $\mathcal{X}$  such that  $\nu$  has an lower and upper bounded density w.r.t.  $\mu$ . Then,  $\lambda_i(T_{k,\nu}) = \Theta(\lambda_i(T_{k,\mu}))$ .
Proof. Let  $p$  be such an upper bounded density, that is,  $\mathrm{d}\nu = p\mathrm{d}\mu$  and there exist  $c, C > 0$  such that  $c \leq p(\pmb{x}) \leq C$  for all  $\pmb{x} \in \mathcal{X}$ . For  $f \in L_2(\nu)$ , we have
$$
\| p \cdot f \| _ {L _ {2} (\mu)} ^ {2} = \int f ^ {2} p ^ {2} \mathrm {d} \mu \leq C \int f ^ {2} p \mathrm {d} \mu = C \int f ^ {2} \mathrm {d} \nu = C \| f \| _ {L _ {2} (\nu)} ^ {2}.
$$
Hence, the linear operator
$$
A: L _ {2} (\nu) \to L _ {2} (\mu), f \mapsto p \cdot f
$$
is well-defined and continuous. It is also easily verified that  $A$  is bijective. Moreover, we have
$$
\langle A f, A f \rangle_ {L _ {2} (\mu)} = \int f ^ {2} p ^ {2} \mathrm {d} \mu \geq c \int f ^ {2} p \mathrm {d} \mu = c \int f ^ {2} \mathrm {d} \nu = c \langle f, f \rangle_ {L _ {2} (\nu)}.
$$
$$
\langle f, T _ {k, \nu} f \rangle_ {L _ {2} (\nu)} = \int \int p (\boldsymbol {x}) f (\boldsymbol {x}) k \left(\boldsymbol {x}, \boldsymbol {x} ^ {\prime}\right) f \left(\boldsymbol {x} ^ {\prime}\right) p \left(\boldsymbol {x} ^ {\prime}\right) \mathrm {d} \mu (\boldsymbol {x}) \mathrm {d} \mu \left(\boldsymbol {x} ^ {\prime}\right) = \left\langle A f, T _ {k, \mu} A f \right\rangle_ {L _ {2} (\mu)}.
$$
Since  $T_{k,\mu}$  and  $T_{k,\nu}$  are compact, self-adjoint, and positive, we can use the Courant-Fischer minmax principle for operators (see e.g. Bell, 2014) to obtain
$$
\begin{array}{l} \lambda_{i}(T_{k,\nu}) = \max_{\substack{V\subseteq L_{2}(\nu)\text{subspace}\\ \dim V = i}}\min_{f\in V\setminus \{0\}}\frac{\langle f,T_{k,\nu}f\rangle_{L_{2}(\nu)}}{\langle f,f\rangle_{L_{2}(\nu)}} \\ \geq c\max_{\substack{V\subseteq L_{2}(\nu)\text{subspace}\\ \dim V = i}}\min_{f\in V\setminus \{0\}}\frac{\langle Af,T_{k,\mu}Af\rangle_{L_{2}(\mu)}}{\langle Af,Af\rangle_{L_{2}(\mu)}} \\ = c\max_{\substack{\tilde{V}\subseteq L_{2}(\nu)\text{subspace}\\ \dim \tilde{V} = i}}\min_{g\in \tilde{V}\setminus \{0\}}\frac{\langle g,T_{k,\mu}g\rangle_{L_{2}(\mu)}}{\langle g,g\rangle_{L_{2}(\mu)}} \\ = c \lambda_ {i} \left(T _ {k, \mu}\right). \\ \end{array}
$$
Here, we have used that since  $A$  is bijective, the subspaces  $AV$  for  $\dim(V) = i$  are exactly the  $i$ -dimensional subspaces of  $L_{2}(\mu)$ . Our calculation above shows that  $\lambda_{i}(T_{k,\mu}) \leq O(\lambda_{i}(T_{k,\nu}))$ . Since  $\mathrm{d}\mu = \frac{1}{p} \mathrm{d}\nu$  with the lower and upper bounded density  $1/p$ , we can reverse the roles of  $\nu$  and  $\mu$  to also obtain  $\lambda_{i}(T_{k,\nu}) \leq O(\lambda_{i}(T_{k,\mu}))$ , which proves the claim.
Lemma F.10 (Integral operators of equivalent kernels have equivalent eigenvalues). Let  $k, \tilde{k} : \mathcal{X} \times \mathcal{X} \to \mathbb{R}$  be bounded kernels with RKHSs  $\mathcal{H}$  and  $\tilde{\mathcal{H}}$  satisfying  $\mathcal{H} = \tilde{\mathcal{H}}$  as sets. Moreover, let  $C \geq 1$  be a constant such that the corresponding RKHS-norms are  $C$ -equivalent and let  $\nu$  be a finite measure on  $\mathcal{X}$ . If there exist constants  $q > 0$  and  $c > 0$  with
$$
\lambda_ {i} \left(T _ {k, \nu}\right) \leq c i ^ {- q}, \quad \quad \quad \quad i \geq 1,
$$
then we also have
$$
\lambda_ {i} (T _ {\tilde {k}, \nu}) \leq c \cdot C ^ {2} \cdot K _ {q} \cdot i ^ {- q}, \quad \quad \quad i \geq 1,
$$
where  $K_{q} > 0$  is a constant only depending on  $q$ .
Proof. We follow the ideas outlined in (Steinwart, 2017, Section 3). To this end, let  $I_{k,\nu} : \mathcal{H} \to L_2(\nu)$  be the embedding  $h \mapsto [h]_{\sim}$ , which is defined and compact since  $k$  is bounded and  $\nu$  is finite, see e.g. (Steinwart and Scovel, 2012, Lemma 2.3). We write  $S_{k,\nu} := I_{k,\nu}^{*}$  for its adjoint, which in turn gives  $I_{k,\nu} = S_{k,\nu}^{*}$ . Then (Steinwart and Scovel, 2012, Lemma 2.2) shows  $T_{k,\nu} = S_{k,\nu}^{*} \circ S_{k,\nu}$ . We denote the  $i$ -th (dyadic) entropy number of  $I_{k,\nu}$  by  $\varepsilon_i(I_{k,\nu})$ , see e.g. Carl and Stephani (1990), (Edmunds and Triebel, 1996, Chapter 1.3.1), or (Steinwart and Christmann, 2008, Chapter 6) for
a definition<sup>3</sup>. Moreover, we denote the  $i$ -approximation and singular numbers of  $I_{k,\nu}$  by  $a_i(I_{k,\nu})$  respectively  $s_i(I_{k,\nu})$ . Since  $I_{k,\nu}$  compactly acts between Hilbert spaces, we then have  $a_i(I_{k,\nu}) = s_i(I_{k,\nu})$ , see e.g. (Pietsch, 1987, Chapter 2.11). This implies
$$
\lambda_ {i} \left(T _ {k, \nu}\right) = a _ {i} ^ {2} \left(I _ {k, \nu}\right) \tag {F.8}
$$
for all  $i \geq 1$  by the very definition of singular numbers. Finally, analogous definitions and considerations are made for the kernel  $\tilde{k}$ . From  $C^{-1} \| \cdot \|_{\mathcal{H}} \leq \| \cdot \|_{\tilde{\mathcal{H}}} \leq C \| \cdot \|_{\mathcal{H}}$  we then conclude that
$$
C ^ {- 1} \varepsilon_ {i} \left(I _ {k, \nu}\right) \leq \varepsilon_ {i} \left(I _ {\tilde {k}, \nu}\right) \leq C \varepsilon_ {i} \left(I _ {k, \nu}\right), \quad i \geq 1, \tag {F.9}
$$
by the multiplicativity of entropy numbers, see e.g. (Edmunds and Triebel, 1996, Chapter 1.3.1).
Now, (F.8) and our eigenvalue assumption yield
$$
a _ {i} \left(I _ {k, \nu}\right) \leq \sqrt {c} \cdot i ^ {- q / 2}, \quad i \geq 1,
$$