SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \Psi \Psi^ {\top} \mathbf {P} _ {> M} = \Psi \Psi^ {\top} \Lambda \Psi \mathbf {R} \Psi_ {> M} ^ {\top} \\ = \boldsymbol {\Psi} \mathbf {K} (\mathbf {K} + \lambda N \mathbf {I} _ {M}) ^ {- 1} \boldsymbol {\Psi} _ {> M} ^ {\top} \\ = \boldsymbol {\Psi} \left(\mathbf {I} _ {M} - \lambda N (\mathbf {K} + \lambda N \mathbf {I} _ {M}) ^ {- 1}\right) \boldsymbol {\Psi} _ {> M} ^ {\top} \\ = \boldsymbol {\Psi} \boldsymbol {\Psi} _ {> M} ^ {\top} - \lambda N \boldsymbol {\Psi} (\mathbf {K} + \lambda N \mathbf {I} _ {M}) ^ {- 1} \boldsymbol {\Psi} _ {> M} ^ {\top}. \\ \end{array}
$$
Analogously to the computations in (24)-(25)
$$
\begin{array}{l} (\mathbf {I} _ {M} + \boldsymbol {\Delta}) \mathbf {P} _ {> M} = \boldsymbol {E} - \lambda \boldsymbol {\Psi} (\mathbf {K} + \lambda N \mathbf {I} _ {M}) ^ {- 1} \boldsymbol {\Psi} _ {> M} ^ {\top} \\ (\mathbf {I} _ {M} + \boldsymbol {\Delta}) \mathbf {P} _ {> M} = \boldsymbol {E} - \lambda \boldsymbol {\Lambda} ^ {- 1} \boldsymbol {\Lambda} \boldsymbol {\Psi} (\mathbf {K} + \lambda N \mathbf {I} _ {M}) ^ {- 1} \boldsymbol {\Psi} _ {> M} ^ {\top} \\ (\mathbf {I} _ {M} + \boldsymbol {\Delta}) \mathbf {P} _ {> M} = \boldsymbol {E} - \lambda \boldsymbol {\Lambda} ^ {- 1} \mathbf {P} _ {> M} \\ (\boldsymbol {\Lambda} + \boldsymbol {\Lambda} \boldsymbol {\Delta}) \mathbf {P} _ {> M} = \boldsymbol {\Lambda} \boldsymbol {E} - \lambda \mathbf {P} _ {> M} \\ (\boldsymbol {\Lambda} + \boldsymbol {\Lambda} \boldsymbol {\Delta} + \lambda \mathbf {I} _ {M}) \mathbf {P} _ {> M} = \boldsymbol {\Lambda} \boldsymbol {E} \\ \mathbf {P} _ {> M} = \left(\boldsymbol {\Lambda} + \boldsymbol {\Lambda} \boldsymbol {\Delta} + \lambda \mathbf {I} _ {M}\right) ^ {- 1} \boldsymbol {\Lambda} \boldsymbol {E} \\ \mathbf {P} _ {> M} = \mathbf {B} E. \\ \end{array}
$$
Lemma C.4 (Fitting Error). Recall the notation
$$
f i t t i n g \quad e r r o r = \| \tilde {\gamma} - \mathbf {P} _ {\leq M} \tilde {\gamma} - \tilde {\gamma} _ {> M} \mathbf {P} _ {> M} \| _ {2} ^ {2},
$$
$$
\mathbf {B} \stackrel {\text {d e f .}} {=} \left(\mathbf {I} _ {M} + \boldsymbol {\Delta} + \lambda \boldsymbol {\Lambda} ^ {- 1}\right) ^ {- 1}.
$$
We have fitting error  $= \left\| \mathbf{B}\left(\lambda \boldsymbol{\Lambda}^{-1}\tilde{\boldsymbol{\gamma}} -\boldsymbol {E}\tilde{\boldsymbol{\gamma}}_{>M}\right)\right\| _2^2$
Proof. By lemmata C.2 and C.3,
$$
\begin{array}{l} \left\| \tilde {\gamma} - \mathbf {P} _ {\leq M} \tilde {\gamma} - \tilde {\gamma} _ {> M} \mathbf {P} _ {> M} \right\| _ {2} ^ {2} = \left\| \tilde {\gamma} - \left(\mathbf {I} _ {M} - \lambda \mathbf {B} \boldsymbol {\Lambda} ^ {- 1}\right) \tilde {\gamma} \tilde {\gamma} - \mathbf {B} \boldsymbol {E} \tilde {\gamma} _ {> M} \right\| _ {2} ^ {2} \\ = \left\| \mathbf {B} \left(\lambda \boldsymbol {\Lambda} ^ {- 1} \tilde {\boldsymbol {\gamma}} - \boldsymbol {E} \tilde {\boldsymbol {\gamma}} _ {> M}\right) \right\| _ {2} ^ {2}. \\ \end{array}
$$
Hence we come up with a new expression of the bias:
Proposition C.5 (Bias). Recall that  $\mathbf{B} \stackrel{\mathrm{def.}}{=} (\mathbf{I}_M + \pmb{\Delta} + \lambda \pmb{\Lambda}^{-1})^{-1}$ . The bias  $\mathbb{E}_x\left(f_{\mathbf{X}}^\lambda(x) - \tilde{f}(x)\right)^2$  has the following expression:
$$
b i a s = \tilde {\gamma} _ {> M} ^ {2} + \left\| \mathbf {B} \left(\lambda \boldsymbol {\Lambda} ^ {- 1} \tilde {\gamma} - \tilde {\gamma} _ {> M} \boldsymbol {E}\right) \right\| _ {2} ^ {2}.
$$
Proof. We apply Proposition C.1 and Lemma C.4 to obtain the result.
C.1.2 Variance
If we consider noise in the label, we have to compute the variance part of the test error.
Proposition C.6 (Variance Expression). Define
$$
\begin{array}{l} \mathbf {M} \stackrel {\text {d e f .}} {=} \mathbb {E} _ {x} \left[ \mathbf {K} _ {x} \mathbf {K} _ {x} ^ {\top} \right] \\ = \mathbb {E} _ {x} \left[ \boldsymbol {\Psi} ^ {\top} \boldsymbol {\Lambda} \boldsymbol {\psi} (x) \boldsymbol {\psi} (x) ^ {\top} \boldsymbol {\Lambda} \boldsymbol {\Psi} \right] \\ = \boldsymbol {\Psi} ^ {\top} \boldsymbol {\Lambda} \mathbb {E} _ {x} [ \psi (x) \psi (x) ^ {\top} ] \boldsymbol {\Lambda} \boldsymbol {\Psi} \\ = \boldsymbol {\Psi} ^ {\top} \boldsymbol {\Lambda} \mathbf {I} _ {M} \boldsymbol {\Lambda} \boldsymbol {\Psi} \\ = \boldsymbol {\Psi} ^ {\top} \boldsymbol {\Lambda} ^ {2} \boldsymbol {\Psi}. \\ \end{array}
$$
We can further simplify the variance part:
$$
\begin{array}{l} \text {v a r i a n c e} \stackrel {\text {d e f .}} {=} \mathbb {E} _ {x, \varepsilon} \left[ \left(\mathbf {K} _ {x} ^ {\top} \mathbf {R} \varepsilon\right) ^ {2} \right] \\ = \mathbb {E} _ {x, \varepsilon} \left[ \boldsymbol {\varepsilon} ^ {\top} \mathbf {R} \mathbf {K} _ {x} \mathbf {K} _ {x} ^ {\top} \mathbf {R} \boldsymbol {\varepsilon} \right] \\ = \mathbb {E} _ {\varepsilon} \left[ \boldsymbol {\varepsilon} ^ {\top} \mathbf {R M R} \boldsymbol {\varepsilon} \right] \\ = \sigma^ {2} \cdot \operatorname {T r} [ \mathbf {R M R} ]. \\ \end{array}
$$
Theorem C.7 (Variance). Recall that  $\mathbf{B} \stackrel{\mathrm{def.}}{=} (\mathbf{I}_M + \pmb{\Delta} + \lambda \pmb{\Lambda}^{-1})^{-1}$ . The variance part, variance, can be expressed as:
$$
\operatorname {v a r i a n c e} = \frac {\sigma^ {2}}{N} \operatorname {T r} \left[ \mathbf {B} ^ {2} \left(\mathbf {I} _ {M} + \boldsymbol {\Delta}\right) \right].
$$
Proof. We argue similarly as in lemma C.2. Since
$$
\begin{array}{l} \Psi \Psi^ {\top} \Lambda \Psi \mathbf {R} = \Psi \mathbf {K} (\mathbf {K} + \lambda N \mathbf {I} _ {M}) ^ {- 1} \\ = \boldsymbol {\Psi} (\mathbf {I} _ {M} - \lambda N \mathbf {R}) \\ = \boldsymbol {\Psi} - \lambda N \boldsymbol {\Psi} \mathbf {R}, \\ \end{array}
$$
therefore, we deduce that
$$
\left(\mathbf {I} _ {M} + \boldsymbol {\Delta}\right) \boldsymbol {\Lambda} \boldsymbol {\Psi} \mathbf {R} = \frac {1}{N} \boldsymbol {\Psi} - \lambda \boldsymbol {\Psi} \mathbf {R} \tag {26}
$$
$$
\left(\mathbf {I} _ {M} + \boldsymbol {\Delta}\right) \boldsymbol {\Lambda} \boldsymbol {\Psi} \mathbf {R} = \frac {1}{N} \boldsymbol {\Psi} - \lambda \boldsymbol {\Lambda} ^ {- 1} \boldsymbol {\Lambda} \boldsymbol {\Psi} \mathbf {R}
$$
$$
(\mathbf {I} _ {M} + \boldsymbol {\Delta} + \lambda \boldsymbol {\Lambda} ^ {- 1}) \boldsymbol {\Lambda} \boldsymbol {\Psi} \mathbf {R} = \frac {1}{N} \boldsymbol {\Psi}
$$
$$
\boldsymbol {\Lambda} \boldsymbol {\Psi} \mathbf {R} = \frac {1}{N} \left(\mathbf {I} _ {M} + \boldsymbol {\Delta} + \lambda \boldsymbol {\Lambda} ^ {- 1}\right) ^ {- 1} \boldsymbol {\Psi}
$$
$$
\boldsymbol {\Lambda} \boldsymbol {\Psi} \mathbf {R} = \frac {1}{N} \mathbf {B} \boldsymbol {\Psi}. \tag {27}
$$
By leveraging the identity  $\mathbf{M} = \boldsymbol{\Psi}^{\top}\boldsymbol{\Lambda}^{2}\boldsymbol{\Psi}$  and elementary properties of the trace map, the computations in (26)-(27) imply that
$$
\begin{array}{l} \operatorname {T r} \left[ \mathbf {R M R} \right] = \operatorname {T r} \left[ \mathbf {R} \boldsymbol {\Psi} ^ {\top} \boldsymbol {\Lambda} ^ {2} \boldsymbol {\Psi} \mathbf {R} \right] (28) \\ = \operatorname {T r} \left[ \left(\boldsymbol {\Lambda} \boldsymbol {\Psi} \mathbf {R}\right) ^ {\top} \left(\boldsymbol {\Lambda} \boldsymbol {\Psi} \mathbf {R}\right) \right] (29) \\ = \operatorname {T r} \left[ \left(\frac {1}{N} \mathbf {B} \boldsymbol {\Psi}\right) ^ {\top} \left(\frac {1}{N} \mathbf {B} \boldsymbol {\Psi}\right) \right] (30) \\ = \frac {1}{N} \operatorname {T r} \left[ \frac {1}{N} \boldsymbol {\Psi} ^ {\top} \mathbf {B} ^ {\top} \mathbf {B} \boldsymbol {\Psi} \right] \\ = \frac {1}{N} \operatorname {T r} \left[ \mathbf {B} ^ {\top} \mathbf {B} \cdot \frac {1}{N} \boldsymbol {\Psi} \boldsymbol {\Psi} ^ {\top} \right] (31) \\ = \frac {1}{N} \operatorname {T r} \left[ \mathbf {B} ^ {\top} \mathbf {B} \left(\mathbf {I} _ {M} + \boldsymbol {\Delta}\right) \right] (32) \\ = \frac {1}{N} \operatorname {T r} \left[ \mathbf {B} ^ {2} \left(\mathbf {I} _ {M} + \boldsymbol {\Delta}\right) \right]; (33) \\ \end{array}
$$
in more detail: in line (28), we use the definition of  $\mathbf{M}$ ; in line (29), we use the fact that both  $\pmb{\Lambda}$  and  $\mathbf{R}$  are symmetric; in line (30), we use line (27); in line (31), we use the cyclicity of the trace; in line (32), we use the definition of  $\pmb{\Delta}$ ; in line (33), we use the symmetry of  $\mathbf{B}$ . We obtain the result upon applying Lemma C.6.
C.1.3 Test Error
The Bias-Variance trade-off (see Proposition B.5) decomposed the KRR's test error into two terms, the bias and variance. Since Propositions C.5 and C.7 give us exact expressions for the bias and variance, respectively, we deduce the following exact expression for the KRR's test error.
Theorem C.8 (Exact Formula for KRR's Test Error). The test error  $\mathcal{R}_{\mathbf{Z},\lambda}$  of KRR equals
$$
\mathcal {R} _ {\mathbf {Z}, \lambda} = \underbrace {\left\| \mathbf {B} \left(\lambda \boldsymbol {\Lambda} ^ {- 1} \tilde {\gamma} - \tilde {\gamma} _ {> M} \boldsymbol {E} _ {M}\right) \right\| _ {2} ^ {2}} _ {\text {b i a s}} + \underbrace {\widehat {\tilde {\gamma} _ {> M} ^ {2}}} _ {\text {b i a s}} + \underbrace {\frac {\sigma_ {n o i s e} ^ {2}}{N} \operatorname {T r} \left[ \mathbf {B} ^ {2} (\mathbf {I} _ {M} + \boldsymbol {\Delta}) \right]} _ {\text {v a r i a n c e}},
$$
where  $\mathbf{B}\stackrel {\mathrm{def.}}{=}(\mathbf{I}_M + \pmb {\Delta} + \lambda \pmb{\Lambda}^{-1})^{-1}$
Proof. We begin with the bias/variance decomposition:
$$
\begin{array}{l} R _ {\mathbf {Z}} ^ {\lambda} \stackrel {\mathrm {d e f .}} {=} \mathbb {E} _ {y} \| f _ {\mathbf {Z}} ^ {\lambda} - \tilde {f} \| _ {L _ {\rho , \chi} ^ {2}} ^ {2} \\ = \mathbb {E} _ {x, y} \left(\mathbf {K} _ {x} ^ {\top} \mathbf {R} \mathbf {y} - \tilde {f} (x)\right) ^ {2} \\ = \mathbb {E} _ {\varepsilon , x} \left(\mathbf {K} _ {x} ^ {\top} \mathbf {R} (\tilde {f} (\mathbf {X}) + \varepsilon) - \tilde {f} (x)\right) ^ {2} \\ = \mathbb {E} _ {x} \left(f _ {\mathbf {X}} ^ {\lambda} (x) - \tilde {f} (x)\right) ^ {2} + \mathbb {E} _ {x, \varepsilon} \left[ \left(\mathbf {K} _ {x} ^ {\top} \mathbf {R} \boldsymbol {\varepsilon}\right) ^ {2} \right] \\ = \text {b i a s} + \text {v a r i a n c e}, \\ \end{array}
$$
then we apply Propositions C.5 and C.7.
For the validation of the Theorem C.8, please see Appendix D for details.
The matrix  $\mathbf{B}$  plays an important role in the expression since it encodes most information of the KRR. Therefore, the following subsection will discuss the approximation of the matrix  $\mathbf{B}$ .
C.2 Matrix Approximation
Recall that the matrix  $\mathbf{B} \stackrel{\mathrm{def.}}{=} (\mathbf{I}_M + \pmb{\Delta} + \lambda \pmb{\Lambda}^{-1})^{-1}$  is the inverse of a random matrix. The following lemma helps to approximate  $\mathbf{B}$ . Informally, it says that: given that  $\delta \stackrel{\mathrm{def.}}{=} \| \pmb{\Delta} \|_{\mathrm{op}} < 1$ . We have
$$
\mathbf {B} = \sum_ {s = 0} ^ {\infty} (- \bar {\mathbf {P}} \Delta) ^ {s} \bar {\mathbf {P}}
$$
in operator norm  $\| \cdot \|_{op}$  for an  $M\times M$  matrix  $\bar{P}$  depending only on the  $M$  eigenvalues  $\{\lambda_k\}_{k = 1}^M$  and on the ridge  $\lambda >0$ . More precisely we have the following.
Lemma C.9 (B-Expansion). Given that  $\delta \stackrel{\mathrm{def.}}{=} \| \pmb{\Delta} \|_{op} < 1$ . It holds that
$$
\lim  _ {n \uparrow \infty} \left\| \mathbf {B} - \sum_ {s = 0} ^ {n} (- \bar {\mathbf {P}} \Delta) ^ {s} \bar {\mathbf {P}} \right\| _ {o p} = 0
$$
where  $\bar{\mathbf{P}}\stackrel {\mathrm{def.}}{=}\mathrm{diag}\left[\frac{\lambda_k}{\lambda_k + \lambda}\right]_k = \pmb {\Lambda}(\pmb {\Lambda} + \lambda \mathbf{I}_M)^{-1}\in \mathbb{R}^{M\times M}.$
Proof. Set  $\mathbf{A} = \mathbf{I}_M + \lambda \mathbf{A}^{-1}$  and repeatedly use the formula  $(\mathbf{A} + \pmb{\Delta})^{-1} = \mathbf{A}^{-1} - \mathbf{A}^{-1}\pmb{\Delta}(\mathbf{A} + \pmb{\Delta})^{-1}$  from [31], we have
$$
\begin{array}{l} \mathbf {B} \stackrel {\text {d e f .}} {=} \left(\mathbf {I} _ {M} + \boldsymbol {\Delta} + \lambda \boldsymbol {\Lambda} ^ {- 1}\right) ^ {- 1} \\ = (\mathbf {A} + \boldsymbol {\Delta}) ^ {- 1} \\ = \mathbf {A} ^ {- 1} - \mathbf {A} ^ {- 1} \boldsymbol {\Delta} (\mathbf {A} + \boldsymbol {\Delta}) ^ {- 1} \\ = \mathbf {A} ^ {- 1} - \mathbf {A} ^ {- 1} \boldsymbol {\Delta} \left(\mathbf {A} ^ {- 1} - \mathbf {A} ^ {- 1} \boldsymbol {\Delta} (\mathbf {A} + \boldsymbol {\Delta}) ^ {- 1}\right) \\ = \mathbf {A} ^ {- 1} - \mathbf {A} ^ {- 1} \boldsymbol {\Delta} \mathbf {A} ^ {- 1} + (\mathbf {A} ^ {- 1} \boldsymbol {\Delta}) ^ {2} (\mathbf {A} + \boldsymbol {\Delta}) ^ {- 1} \\ = \sum_ {s = 0} ^ {n} (- \mathbf {A} ^ {- 1} \boldsymbol {\Delta}) ^ {s} \mathbf {A} ^ {- 1} + (- \mathbf {A} ^ {- 1} \boldsymbol {\Delta}) ^ {n + 1} (\mathbf {A} + \boldsymbol {\Delta}) ^ {- 1} \\ \end{array}
$$
Note that  $\mathbf{A}^{-1} = (\mathbf{I}_M + \lambda \mathbf{\Lambda}^{-1})^{-1} = \mathbf{\Lambda}(\mathbf{\Lambda} + \lambda \mathbf{I}_M)^{-1} = \bar{\mathbf{P}}$  with operator norm  $\frac{\lambda_1}{\lambda_1 + \lambda} < 1$ , hence we have  $(\mathbf{A}^{-1}\pmb {\Delta})^{n + 1} = (-\bar{\mathbf{P}}\pmb {\Delta})^{n + 1}\rightarrow 0$  in operator norm as  $n\to \infty$ . Hence
$$
\begin{array}{l} \mathbf {B} = \sum_ {s = 0} ^ {\infty} (- \mathbf {A} ^ {- 1} \boldsymbol {\Delta}) ^ {s} \mathbf {A} ^ {- 1} \\ = \sum_ {s = 0} ^ {\infty} (- \bar {\mathbf {P}} \Delta) ^ {s} \bar {\mathbf {P}} \\ \end{array}
$$
in operator norm.
Due to the convergence result in lemma C.9, it is natural to define:
Definition C.10. For any  $n \in \mathbb{N} \cup \{\infty\}$ , write  $\mathbf{B}^{(n)} = \sum_{s=0}^{n} (-\bar{\mathbf{P}} \Delta)^{s} \bar{\mathbf{P}}$ . For example, We have
$$
\mathbf {B} ^ {(0)} = \bar {\mathbf {P}}
$$
$$
\mathbf {B} ^ {(1)} = \bar {\mathbf {P}} - \bar {\mathbf {P}} \Delta \bar {\mathbf {P}}
$$
$$
\mathbf {B} ^ {(\infty)} = \mathbf {B}
$$
Although lemma C.9 is valid when  $\delta < 1$ , we need a slightly stronger condition that  $\delta$  is upper bounded by an arbitrary constant strictly small than 1. For simplicity, we assume this constant to be  $\frac{1}{2}$  in the following lemma:
Lemma C.11 (B-Approximation). Assume that  $\delta \stackrel{\mathrm{def.}}{=} \| \pmb{\Delta} \|_{op} < \frac{1}{2}$ . Let  $\mathbf{B}^{(n)} = \sum_{s=0}^{n} (-\bar{\mathbf{P}} \pmb{\Delta})^s \bar{\mathbf{P}}$  be the  $n$ th-order approximation of the matrix  $\mathbf{B}$  as in definition C.10. Then we have
$$
\left\| \mathbf {B} - \mathbf {B} ^ {(n)} \right\| _ {o p} <   2 \delta^ {n + 1}.
$$
Proof. We first bound the operator norm of the matrix  $\mathbf{B}$ : since the minimum singular value of the matrix  $\bar{\mathbf{P}}^{-1} + \Delta$  is at least
$$
\frac {\lambda_ {k} + \lambda}{\lambda_ {k}} - \| \Delta \| _ {\mathrm {o p}} \geq 1 + \frac {\lambda}{\lambda_ {k}} - \frac {1}{2} > \frac {1}{2},
$$
and hence
$$
\left\| \mathbf {B} \right\| _ {\mathrm {o p}} = \left\| \left(\bar {\mathbf {P}} ^ {- 1} + \boldsymbol {\Delta}\right) ^ {- 1} \right\| _ {\mathrm {o p}} <   2.
$$
$$
\begin{array}{l} \mathbf {B} - \mathbf {B} ^ {(n)} = \sum_ {s = n + 1} ^ {\infty} (- \bar {\mathbf {P}} \boldsymbol {\Delta}) ^ {s} \bar {\mathbf {P}} \\ = (- \bar {\mathbf {P}} \Delta) ^ {n + 1} \sum_ {s = 0} ^ {\infty} (- \bar {\mathbf {P}} \Delta) ^ {s} \bar {\mathbf {P}} \\ = (- \bar {\mathbf {P}} \boldsymbol {\Delta}) ^ {n + 1} \mathbf {B}. \\ \end{array}
$$
Hence  $\left\| \mathbf{B} - \mathbf{B}^{(n)}\right\|_{\mathrm{op}} \leq \left\| \bar{\mathbf{P}}\boldsymbol{\Delta}\right\|_{\mathrm{op}}^{n + 1}\left\| \mathbf{B}\right\|_{\mathrm{op}} < \left\| \boldsymbol{\Delta}\right\|_{\mathrm{op}}^{n + 1} \cdot 2 = 2\delta^{n + 1}$ , since we have  $\left\| \bar{\mathbf{P}}\right\|_{\mathrm{op}} = \frac{\lambda_1}{\lambda_1 + \lambda} < 1$ .
Note that the upper bound  $\frac{1}{2}$  of  $\delta$  can be replaced by any constant strictly small than 1 to get a similar conclusion.
Remark C.12. Using the concentration result from random matrix theory, for  $M < N$ , one can show with high probability that the operator norm  $\delta$  of the fluctuation matrix  $\Delta$  is less than 1.
See subsection C.3 for details. Then we can use the above lemmata C.9 and C.11 to approximate the test error of KRR:
Proposition C.13 (Bias Approximation). Fix a sample  $\mathbf{Z}$  of  $\rho$  such that  $\delta \stackrel{\mathrm{def.}}{=} \| \pmb{\Delta} \|_{op} < \frac{1}{2}$ . Then the bias test term is bounded above and below by
$$
\left| \text {b i a s} - \left(\left\| \bar {\mathbf {P}} w \right\| _ {2} ^ {2} + \tilde {\gamma} _ {> M} ^ {2}\right) \right| \leq 2 \delta \left\| \bar {\mathbf {P}} w \right\| _ {2} ^ {2} + \| w \| _ {2} ^ {2} \delta^ {2} p (\delta),
$$
where  $\bar{\mathbf{P}}\stackrel {\mathrm{def.}}{=}\Lambda (\pmb {\Lambda} + \lambda \mathbf{I}_M)^{-1},w = \lambda \pmb{\Lambda}^{-1}\tilde{\gamma} -\tilde{\gamma}_{>M}\pmb {E},$  and  $p(\delta)\stackrel {\mathrm{def.}}{=}5 + 4\delta +4\delta^2$  . By writing  $\pmb {E} = (\eta_k)_{k = 1}^M$  , the bounds simplify to
$$
\text {b i a s} \leq \tilde {\gamma} _ {> M} ^ {2} + (1 + 2 \delta) \sum_ {k = 1} ^ {M} \frac {\left(\lambda \tilde {\gamma} _ {k} - \tilde {\gamma} _ {> M} \eta_ {k} \lambda_ {k}\right) ^ {2}}{\left(\lambda_ {k} + \lambda\right) ^ {2}} + \| w \| _ {2} ^ {2} \delta^ {2} p (\delta);
$$
$$
\text {b i a s} \geq \tilde {\gamma} _ {> M} ^ {2} + (1 - 2 \delta) \sum_ {k = 1} ^ {M} \frac {\left(\lambda \tilde {\gamma} _ {k} - \tilde {\gamma} _ {> M} \eta_ {k} \lambda_ {k}\right) ^ {2}}{\left(\lambda_ {k} + \lambda\right) ^ {2}} - \| w \| _ {2} ^ {2} \delta^ {2} p (\delta).
$$
From there, we differentiate the approach from Bach [6]: From Propositions C.5 and C.7, it is inevitable to approximate the matrix  $\mathbf{B}$ , and we have  $\mathbf{I}_M$  as support of the inverse. Bach instead uses RHKS basis to express the fluctuation matrix and is hence forced to use  $\lambda \mathbf{I}_M$  as the support. As a result, he would need to require that the fluctuation is less than  $\lambda$  and hence his requirement on  $N$  is antiproportional to  $\lambda$  in Theorem 5.1.
Proof. Let  $w = \lambda \pmb{\Lambda}^{-1}\tilde{\gamma} -\tilde{\gamma}_{>M}\pmb{E}$ . We apply lemma C.4 followed by the 1st-order approximation  $\mathbf{B}^{(1)}$  of the matrix  $\mathbf{B}$  in lemma C.11:
fitting error  $= \| \mathbf{B}w\| _2^2 = \left\| \mathbf{B}^{(1)}w + \left(\mathbf{B} - \mathbf{B}^{(1)}\right)w\right\| _2^2$
$$
\begin{array}{l} = \left\| \mathbf {B} ^ {(1)} w \right\| _ {2} ^ {2} + w ^ {\top} \left(\mathbf {B} - \mathbf {B} ^ {(1)}\right) \mathbf {B} ^ {(1)} w + w ^ {\top} \mathbf {B} ^ {(1)} \left(\mathbf {B} - \mathbf {B} ^ {(1)}\right) w + \left\| \left(\mathbf {B} - \mathbf {B} ^ {(1)}\right) w \right\| _ {2} ^ {2} \\ \leq \left\| \mathbf {B} ^ {(1)} w \right\| _ {2} ^ {2} + 2 \left\| \mathbf {B} ^ {(1)} \right\| _ {\mathrm {o p}} \left\| \mathbf {B} - \mathbf {B} ^ {(1)} \right\| _ {\mathrm {o p}} \| w \| _ {2} ^ {2} + \left\| \mathbf {B} - \mathbf {B} ^ {(1)} \right\| _ {\mathrm {o p}} ^ {2} \| w \| _ {2} ^ {2} \\ \leq \left\| \mathbf {B} ^ {(1)} w \right\| _ {2} ^ {2} + 2 \cdot (1 + \delta) \cdot 2 \delta^ {2} \| w \| _ {2} ^ {2} + 4 \delta^ {4} \| w \| _ {2} ^ {2} \\ \leq \left\| \mathbf {B} ^ {(1)} w \right\| _ {2} ^ {2} + 4 \| w \| _ {2} ^ {2} \delta^ {2} (1 + \delta + \delta^ {2}) \\ \leq \left\| (\bar {\mathbf {P}} - \bar {\mathbf {P}} \Delta \bar {\mathbf {P}}) w \right\| _ {2} ^ {2} + 4 \| w \| _ {2} ^ {2} \delta^ {2} (1 + \delta + \delta^ {2}) \\ \leq \left\| \mathbf {I} _ {M} - \bar {\mathbf {P}} \Delta \right\| _ {\mathrm {o p}} ^ {2} \| \bar {\mathbf {P}} w \| _ {2} ^ {2} + 4 \| w \| _ {2} ^ {2} \delta^ {2} (1 + \delta + \delta^ {2}) \\ \leq \left(1 + 2 \| \bar {\mathbf {P}} \boldsymbol {\Delta} \| _ {\mathrm {o p}} + \| \bar {\mathbf {P}} \boldsymbol {\Delta} \| _ {\mathrm {o p}} ^ {2}\right) \| \bar {\mathbf {P}} w \| _ {2} ^ {2} + 4 \| w \| _ {2} ^ {2} \delta^ {2} (1 + \delta + \delta^ {2}) \\ \leq \left(1 + 2 \| \bar {\mathbf {P}} \boldsymbol {\Delta} \| _ {\mathrm {o p}}\right) \| \bar {\mathbf {P}} w \| _ {2} ^ {2} + \| w \| _ {2} ^ {2} \delta^ {2} (5 + 4 \delta + 4 \delta^ {2}) \\ \leq (1 + 2 \delta) \| \bar {\mathbf {P}} w \| _ {2} ^ {2} + \| w \| _ {2} ^ {2} \delta^ {2} (5 + 4 \delta + 4 \delta^ {2}). \\ \end{array}
$$
Hence we have the upper bound:
$$
\operatorname {b i a s} \leq \tilde {\gamma} _ {> M} ^ {2} + (1 + 2 \delta) \left\| \bar {\mathbf {P}} w \right\| _ {2} ^ {2} + \| w \| _ {2} ^ {2} \delta^ {2} p (\delta).
$$
We argue similarly for the lower bound using:  $\| \mathbf{A}\|_{\mathrm{op}}\| v\| _2^2\geq v^\top \mathbf{A}v\geq -\| \mathbf{A}\|_{\mathrm{op}}\| v\| _2^2$  for any  $\mathbf{A}\in \mathbb{R}^{M\times M}$ $v\in \mathbb{R}^{M\times 1}$
We argue similarly for variance.
Proposition C.14 (Variance Approximation). Fix a sampling  $\mathbf{Z}$  such that  $\delta \stackrel{\mathrm{def.}}{=} \| \pmb{\Delta} \|_{op} < \frac{1}{2}$ . Then we have
$$
\left| \text {v a r i a n c e} - \frac {\sigma^ {2}}{N} \sum_ {k = 1} ^ {M} \frac {\lambda_ {k} ^ {2}}{(\lambda_ {k} + \lambda) ^ {2}} \right| \leq \delta \frac {\sigma^ {2}}{N} \sum_ {k = 1} ^ {M} \frac {\lambda_ {k} ^ {2}}{(\lambda_ {k} + \lambda) ^ {2}} + M \frac {\sigma^ {2}}{N} (1 + \delta) \delta^ {2} p (\delta),
$$
where  $p(\delta)\stackrel {\mathrm{def.}}{=}5 + 4\delta +4\delta^2$  , and  $\sigma^2\stackrel {\mathrm{def.}}{=}\mathbb{E}[\epsilon^2 ]$  is the noise variance.
Proof. Note that  $\operatorname{Tr} \mathbf{A} \leq M\|\mathbf{A}\|_{\mathrm{op}}$  for any matrix  $\mathbf{A} \in \mathbb{R}^{M \times M}$ . Since  $\mathbf{B}^2(\mathbf{I}_M + \boldsymbol{\Delta}) = (\mathbf{B}^{(1)})^2(\mathbf{I}_M + \boldsymbol{\Delta}) + 2\mathbf{B}^{(1)}(\mathbf{B} - \mathbf{B}^{(1)})(\mathbf{I}_M + \boldsymbol{\Delta}) + (\mathbf{B} - \mathbf{B}^{(1)})^2(\mathbf{I}_M + \boldsymbol{\Delta})$ , we can bound the residue term by  $\delta$ :
$$
\begin{array}{l} \operatorname {T r} \left[ 2 \mathbf {B} ^ {(1)} (\mathbf {B} - \mathbf {B} ^ {(1)}) (\mathbf {I} _ {M} + \boldsymbol {\Delta}) + (\mathbf {B} - \mathbf {B} ^ {(1)}) ^ {2} (\mathbf {I} _ {M} + \boldsymbol {\Delta}) \right] \\ \leq M (1 + \delta) \left\| \mathbf {B} - \mathbf {B} ^ {(1)} \right\| _ {\mathrm {o p}} \left(2 \left\| \mathbf {B} ^ {(1)} \right\| _ {\mathrm {o p}} + \left\| \mathbf {B} - \mathbf {B} ^ {(1)} \right\| _ {\mathrm {o p}}\right) \\ \leq M (1 + \delta) \cdot 2 \delta^ {2} (2 (1 + \delta) + 2 \delta^ {2}) \\ \leq 4 M \delta^ {2} (1 + \delta) (1 + \delta + \delta^ {2}), \\ \end{array}
$$
For the main terms, we have
$$
\begin{array}{l} \left. \operatorname {T r} \left[ \left(\mathbf {B} ^ {(1)}\right) ^ {2} \left(\mathbf {I} _ {M} + \boldsymbol {\Delta}\right) \right] \leq \operatorname {T r} \left[ \bar {\mathbf {P}} ^ {2} \right] \cdot \left\| \left(\mathbf {I} _ {M} - \boldsymbol {\Delta} \bar {\mathbf {P}}\right) ^ {2} \left(\mathbf {I} _ {M} + \boldsymbol {\Delta}\right) \right\| _ {\mathrm {o p}} \right. \\ = \operatorname {T r} \left[ \bar {\mathbf {P}} ^ {2} \right] \left\| \mathbf {I} _ {M} + \Delta \left(\mathbf {I} _ {M} - 2 \bar {\mathbf {P}}\right) + \left(\Delta \bar {\mathbf {P}}\right) ^ {2} - 2 \Delta \bar {\mathbf {P}} \Delta + \left(\Delta \bar {\mathbf {P}}\right) ^ {2} \Delta \right\| _ {\mathrm {o p}} \\ \leq \operatorname {T r} \left[ \bar {\mathbf {P}} ^ {2} \right] \left\| \mathbf {I} _ {M} + \Delta \left(\mathbf {I} _ {M} - 2 \bar {\mathbf {P}}\right) \right\| _ {\mathrm {o p}} + M \left\| \left(\Delta \bar {\mathbf {P}}\right) ^ {2} - 2 \Delta \bar {\mathbf {P}} \Delta + \left(\Delta \bar {\mathbf {P}}\right) ^ {2} \Delta \right\| _ {\mathrm {o p}} \\ \leq \operatorname {T r} [ \bar {\mathbf {P}} ^ {2} ] (1 + \delta) + M \delta^ {2} (1 + \delta). \\ \end{array}
$$
We apply Theorem C.7 to yield a bound on variance:
$$
\left| \operatorname {v a r i a n c e} - \frac {\sigma^ {2}}{N} \sum_ {k = 1} ^ {M} \frac {\lambda_ {k} ^ {2}}{(\lambda_ {k} + \lambda) ^ {2}} \right| \leq \delta \frac {\sigma^ {2}}{N} \sum_ {k = 1} ^ {M} \frac {\lambda_ {k} ^ {2}}{(\lambda_ {k} + \lambda) ^ {2}} + M \frac {\sigma^ {2}}{N} (1 + \delta) \delta^ {2} p (\delta).
$$
Note that the above propositions C.13 and C.14 give absolute (non-probabilistic) bounds on the test error, once  $\delta$  is controlled.
C.3 Concentration Results