SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ridge parameter The bounds in Eq. (12) demonstrate that in the ridgeless case, the bias can be controlled solely by the finite rank error  $\tilde{\gamma}_k^2$  with a confidence level depending on  $N$ ; also the upper and lower bounds coincide as  $N \to \infty$ .
Variance For the variance bounds in Eq. (10), we have similar results: the variance can be controlled solely by the (sample-independent) noise variance  $\sigma^2$  with a confidence level depending on  $N$ , also the upper and lower bounds coincide as  $N \to \infty$ .
5.2 Comparison with Prior Works
We discuss how our results add to, and improve on, what is known about finite-rank KRRs following the presentation in Table 1.
Classical tools to study generalization The test error measures the average difference between the trained model and the target function. It is one of the most common measures to study the generalization performance of a machine learning model. Nevertheless, generally applicable statistical learning-theoretic tools from VC-theory [8], (local) Rademacher complexities [22, 7, 30], PAC-Bayes methods [1, 28], or smooth optimal transport theory [18] all yield pessimistic bounds on the KRR test error. For example, Mohri et al. [30] bound the generalization error using Rademacher Complexity: there exists some constant  $C > 0$  independent to  $N$  and  $\lambda$  such that, with a probability of at least  $1 - \tau$ :
$$
\text {t e s t} \leq \frac {C}{\sqrt {N}} \left(1 + \frac {1}{2} \sqrt {\frac {\log \frac {1}{\tau}}{2}}\right). \tag {14}
$$
If we set  $\tau$  to  $2 / N$ , the decay in the generalization gap is  $\mathcal{O}\left(\sqrt{\frac{\log N}{N}}\right)$ , which is too slow compared to other kernel-specific analyses.
Truncated KRR Amini et al. [2] suggests an interesting type of finite-rank kernels: for any (infinite-rank) kernel  $K^{(\infty)}$ , fix a sample  $\mathbf{Z} = (\mathbf{X}, \mathbf{y})$  of size  $N$  and define a rank- $N$  kernel  $K$  by the eigendecomposition of the kernel matrix  $\mathbf{K}^{(\infty)}$ . Note that different random sample results in a completely different rank- $N$  kernel  $K$ . Assume that the target function  $\tilde{f}$  lies in the  $N$ -dimensional RKHS corresponding to  $K$ , then one can obtain an exact formula for the expected value of the test error of the kernel  $K$  (but not only the original  $K^{(\infty)}$ ). However, the formula obtained in [2] only takes into account the expectation over the noise  $\epsilon$ , but not over the samples  $x$ . Hence, our paper yields a more general result for the test error.
Upper bound Comparison To the best of our knowledge, the following result is the closest and most related to ours:
Theorem 5.1. (Proposition 7.4 in [6]) With notation as before, assume, in addition, that:  $1)\tilde{\gamma}_{>M} = 0$  that is,  $\tilde{f}\in \mathcal{H};2)$  for all  $k = 1,\dots,M$ , we have  $\sqrt{\sum_{k = 1}^{M}\lambda_k}\leq R;3)$  and  $N\geq \left(\frac{4}{3} +\frac{R^2}{8\lambda}\log \frac{14R^2}{\lambda\tau}\right)$ .
Then we have, with a probability of at least  $1 - \tau$
$$
b i a s \leq 4 \lambda \| \tilde {f} \| _ {\mathcal {H}} ^ {2};
$$
$$
\text {v a r i a n c e} \leq \frac {8 \sigma^ {2} R ^ {2}}{\lambda N} \left(1 + 2 \log \frac {2}{\tau}\right).
$$
In comparison to [6], Theorem 4.2 makes the following significant improvements:
(i) In Theorem 5.1, as the ridge parameter  $\lambda \rightarrow 0$ , the minimum requirement of  $N$  and the upper bound of the variance explode; while in our case, both the minimum requirement of  $N$  and the bounds on the variance are independent of  $\lambda$ .
(ii) While [6] also mentioned the case where  $\tilde{\gamma}_{>M} \neq 0$ , however their bound for the inconsistent case is implicit; while our work clearly states the impact of  $\tilde{\gamma}_{>M}$  on the test error.
(iii) Our upper bounds are sharper than [6]:
(a) For the bias, assume  $\tilde{\gamma}_{>M} = 0$  for comparison purposes. Then by line (13) our upper bound would become:
$$
\operatorname {b i a s} \leq \lambda \| \tilde {f} \| _ {\mathcal {H}} ^ {2} \left(1 + 2 \sqrt {\frac {\log N}{N}}\right) + C _ {1} \frac {\log N}{N},
$$
which improves [6]'s upper bound bias  $\leq 4\lambda \| \tilde{f}\|_{\mathcal{H}}^2$  significantly. First, we observe a minor improvement in the constant (by a factor of  $\frac{1}{4}$ ) when  $N\to \infty$ . Second and more importantly, for finite  $N$ , we observe a non-asymptotic decay on the upper bound, while [6]'s upper bound is insensitive to  $N$ .
(b) For the variance, if we replace  $\tau$  by  $2 / N$  in [6]'s upper bound: with probability of at least  $1 - 2 / N$ ,
$$
\operatorname {v a r i a n c e} \leq \frac {8 \sigma^ {2} R ^ {2}}{\lambda N} (1 + 2 \log N), \tag {15}
$$
which has a decay rate of  $\frac{\log N}{N}$ , our work shows a much faster decay of  $\frac{1}{N}$ . Moreover, we show that the "unscaled" variance, that is  $N \cdot (\text{variance})$ , is actually bounded by  $\mathcal{O}\left(1 + \sqrt{\frac{\log N}{N}}\right)$ , while [6]'s upper bound on it would explode.
(iv) We also provide a lower bound for the test error that matches with the upper bound when  $N \to \infty$ , while [6] does not derive any lower bound.
While our work offers overall better bounds, there are some technical differences between the bounds of [6] and ours:
(i) We require some explicit mild condition on input distribution  $\rho$  and on kernel  $K$ , while [6] rely on the assumption  $\sqrt{\sum_{k=1}^{M} \lambda_k} \leq R$  which gives an alternative implicit requirement;
(ii) Our work eventually assumes under-parametrization  $M < N$ , while [6] does not require this. Essentially, we exploit the assumption on the under-parameterized regime to obtain a sharper bound than [6]. In short, we select the  $L_{\rho}^{2}$  basis  $\psi_{k}$  for analysis while [6] selects the basis in RKHS, which eventually affects the assumptions and conclusions. For more details, see Appendix C.
Nyström Subsampling and Random Feature Model Nyström Subsampling and Random feature models are two setting closely related to our work. [34, 35] bound the test error from above where the regularizer  $\lambda = \lambda(N)$  decays as  $N \to \infty$ . In comparison, one main contribution of our work is that we provide both tighter upper bound and tighter lower bound than these prior works (they derive the same convergence rate on the upper bound up to constants but they do not derive a lower bound). Another major difference is that our bounds work for any regularization independent to  $N$ .
For inconsistent cases (or misspecified model), [6] only derives an upper bound on the expected value of test error. Also, the bound involves a function defined as an infimum of some set.
Effective Dimension In [34, 35] and other related works on kernel, the quantity  $\mathcal{N}(\lambda) = \sum_{k=1}^{M} \frac{\lambda_k}{\lambda_k + \lambda} = \operatorname{Tr}\left[\bar{\mathbf{P}}\right]$  is called the effective dimension and it appeared as a factor in the upper bound. In our work, we can define a related quantity  $\mathcal{N}^2(\lambda) = \sum_{k=1}^{M} \frac{\lambda_k^2}{(\lambda_k + \lambda)^2} = \operatorname{Tr}\left[\bar{\mathbf{P}}^2\right]$ , which appears in our bound (See Proposition C.14 for details.). Note that  $\mathcal{N}^2(\lambda) \leq \mathcal{N}(\lambda) \leq M$ . Indeed, we can sharpen the factor  $\frac{M}{N}$  in line (10) to  $\frac{\mathcal{N}^2(\lambda)}{N}$ .
5.3 Experiments
We examine the test error and showcase the validity of the bounds we derived for two different finite-rank kernels: the truncated neural tangent kernel (tNTK) and the Legendre kernel (LK) (see details of their definitions in Appendix D). Figure 1 shows the true test error as a function of the sample size  $N$ . Figure 2 plots our upper bound compared to [6], clearly demonstrating significant improvements. We for instance note that the bound is tighter for any value of  $\lambda$ . Finally, Figure 9 shows both the upper and lower bound closely enclose the true test error. For more details on the experiment setup, including evaluating the lower bound, please see Appendix D.
Figure 1: KRR on a finite-rank kernel. Left: KRR training; right: test error for varying  $N$ , over 10 iterations. The upper and lower quartiles are used as error bars.
5.4 Future Research Direction
An interesting extension would be to combine our results with [36, 10] in the over-parameterized regime to give an accurate transition and observe double descent. However, since we are interested in the sharpest possible bounds for the under-parameterized regime, we treat these cases separately. We will do the same for the over-parameterized regime in future work. A special case is where the parameterization is at the threshold  $M = N$  and  $\lambda = 0$ . The blow-up of variance as  $M \to N$  is well-known, where [5, 9] gives empirical and theoretical reports on the blow-up for kernels. We expect that we can formally prove this result for any finite-rank kernels using our exact formula in Proposition C.7 and some anti-concentration inequalities on random matrices.
Figure 2: Comparison of test error bounds. Left: upper bounds for varying  $N$ ; right: upper bounds for varying  $\lambda$ . In natural log scale.
Figure 3: Our bounds (u.b. = upper bound, l.w. = lower bound) comparing to the test error with varying  $N$ , over 10 iterations, in natural log scale. The residues with coefficients  $C_1$  and  $C_2$  are dropped by simplicity. As a result, the bounds are not 'bounding' the averaged test error for small  $N$ . But for large  $N$ , the residues become negligible.
[1] Pierre Alquier, James Ridgway, and Nicolas Chopin. On the properties of variational approximations of Gibbs posteriors. J. Mach. Learn. Res., 17:Paper No. 239, 41, 2016.
[38] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances in neural information processing systems, 29, 2016.
[39] Martin J. Wainwright. High-Dimensional Statistics: A Non-Asymptotic Viewpoint. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2019.
[40] Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? Advances in neural information processing systems, 27, 2014.
A Glossary
Table 2: Glossary
Name	Notation	Expression	Dimension
sampling distribution	ρ	-	X→R+
sampling size	N	-	integer
input matrix	X	(xi)N i=1 ~ iid ρx	N × d
output vector	y	(yi)N i=1	N × 1
sample	Z	(X,y)	N × (d+1)
noise	ε	-	random scalar
noise variance	σ2	E[ε2]	scalar
ridge	λ	-	scalar
finite-rank kernel	K	∑k=1M λkψk(·)ψk(·)	X × X → R
kernel rank	M	-	integer
kth eigenfunction	ψk	-	X → R
kth value	λk	-	scalar
-	ψ(x)	[ψk(x)]k=1	M × 1
-	Ψ	[ψk(xi)]k,i	M × N
-	Λ	diag [λk]	M × M
kernel matrix	K	[K(xi, xj)]i,j = ΨT Λ Ψ	N × N
resolvent	R	(K + λNIN)^{-1}	N × N
target function	f	∑k=1M γkψk + γ>Mψ>M	X → R
-	f≤M	∑k=1M γkψk	X → R
kth target coefficient	γk	∫χ f(x)ψk(x)dρX(x)	scalar
-	γ	[γk]	M × 1
orthonormal complement	ψ>M	-	X → R
complementary coefficient	γ>M	-	scalar
-	Ψ>M	[ψ>M(xi)]	1 × N
test error	RZ,λ	Ex,ε[(fZ,λ(x) - f(x))2]	scalar
bias	-	∫χ (f(x, f(X), λ(x) - f(x))2 dρ(x)	scalar
variance	-	RZ,λ - bias = Ex,ε (KxT Rε)2	scalar
fluctuation matrix	Δ	1/N ΨΨT - IM	M × M
fluctuation	δ	||Δ|op	scalar
error vector	E	[ηk]	M × 1
-	ηk	1/N ∑i=1N ψk(xi)ψ>M(xi)	scalar
-	B	(IM + Δ + λΔ-1)-1	M × M
-	P	diag [λk/λk+λ]	M × M
B Classical KRR Theory
In an effort to keep our manuscript as self-contained as possible, we recall the Mercer decomposition, representer theorem for kernel ridge regression as well as the form of the bias-variance tradeoff in the KRR context.
B.1 Mercer Decomposition
We begin with a general kernel  $K^{(\infty)}:\mathcal{X}\times \mathcal{X}\to \mathbb{R}$
Proposition B.1. [12] Fix a sample distribution  $\rho$ . Let  $K^{(\infty)}: \mathcal{X} \times \mathcal{X} \to \mathbb{R}$  be a reproducing kernel with corresponding RKHS  $\mathcal{H}^{(\infty)}$ . There exists a decreasing sequence of real numbers  $\lambda_1 \geq \lambda_2 \geq \dots$ , called the eigenvalues of the kernel  $K^{(\infty)}$ ; and a sequence of pairwise-orthonormal functions  $\{\psi_k\}_{k=1}^{\infty} \subset L_{\rho}^2$ , called the eigenfunctions of  $K^{(\infty)}$ , such that for all  $x, x' \in \mathcal{X}$ , we have
$$
K ^ {(\infty)} \left(x, x ^ {\prime}\right) = \sum_ {k = 1} ^ {\infty} \lambda_ {k} \psi_ {k} (x) \psi_ {k} \left(x ^ {\prime}\right) \tag {16}
$$
In particular, we assume  $\lambda_{k} = 0$ ,  $\forall k > M$ . In this case, we say the kernel  $K(x,x^{\prime}) = \sum_{k = 1}^{M}\lambda_{k}\psi_{k}(x)\psi_{k}(x^{\prime})$  is of finite rank  $M$  with corresponding (finite-dimensional) RKHS  $\mathcal{H}$ , recovering equation (2).
The first of these results, allows us to explicitly express the finite-rank kernel ridge regressor  $f_{\mathbf{Z},\lambda}$ .
Proposition B.2 (Representer Theorem - [39, Chapter 12]). Let  $\mathbf{R} \stackrel{\mathrm{def.}}{=} (\mathbf{K} + \lambda N\mathbf{I}_N)^{-1} \in \mathbb{R}^{N \times N}$  be the resolvent matrix and recall the kernel ridge regressor  $f_{\mathbf{Z},\lambda}$  given by equation (3):
$$
f_{\mathbf{Z},\lambda}\stackrel {\text{def.}}{=}\arg \min_{f\in \mathcal{H}}\frac{1}{N}\sum_{i = 1}^{N}\left(f(x_{i}) - y_{i}\right)^{2} + \lambda \| f\|_{\mathcal{H}}^{2}
$$
Then, for every  $x \in \mathcal{X}$ , we have the expression
$$
f _ {\mathbf {Z}, \lambda} (x) = \mathbf {y} ^ {\top} \mathbf {R} \mathbf {K} _ {x}, \forall x \in \mathcal {X}, \tag {17}
$$
where  $\mathbf{K}_x\stackrel {\mathrm{def.}}{=}[K(x_i,x)]_{i = 1}^N\in \mathbb{R}^{N\times 1}$
B.2 Compact Matrix Expression
First, let  $\Psi \stackrel{\mathrm{def.}}{=} (\psi_k(x_i))_{k=1,i=1}^{M,N}$  be the random  $M \times N$  matrix defined by evaluating the  $M$  eigenfunctions on all input training instances  $\mathbf{X} \stackrel{\mathrm{def.}}{=} (x_i)_{i=1}^N$ ,  $\Lambda \stackrel{\mathrm{def.}}{=} \operatorname{diag}[\lambda_k] \in \mathbb{R}^{M \times M}$ , and  $\psi(x) \stackrel{\mathrm{def.}}{=} [\psi_k(x)]_{k=1}^M \in \mathbb{R}^{M \times 1}$ . The advantage of this notation is that we can rewrite the equations in a more compact form. For equation (17):
$$
f _ {\mathbf {Z}, \lambda} (x) = \mathbf {y} ^ {\top} \underbrace {\left(\boldsymbol {\Psi} ^ {\top} \boldsymbol {\Lambda} \boldsymbol {\Psi} + \lambda N \mathbf {I} _ {M}\right) ^ {- 1}} _ {\mathbf {R}} \boldsymbol {\Psi} ^ {\top} \boldsymbol {\Lambda} \boldsymbol {\psi} (x); \tag {18}
$$
for equation (4):
$$
\tilde {f} (x) = \tilde {\gamma} ^ {\top} \psi (x) + \tilde {\gamma} _ {> M} \psi_ {> M} (x). \tag {19}
$$
Last but not least, we define some important quantities for later analysis.
Definition B.3 (Fluctuation matrix). The fluctuation matrix is the random  $M \times M$ -matrix given by  $\boldsymbol{\Delta} \stackrel{\mathrm{def.}}{=} \frac{1}{N} \boldsymbol{\Psi} \boldsymbol{\Psi}^{\top} - \mathbf{I}_M$ . Our analysis will often involve the operator norm of  $\boldsymbol{\Delta}$ , which we denote by  $\delta \stackrel{\mathrm{def.}}{=} \| \boldsymbol{\Delta} \|_{\mathrm{op}}$ .
The fluctuation matrix  $\pmb{\Delta}$  measures the first source of randomness in the KRR's test error. Namely it encodes the degree of non-orthonormality between the vectors obtained by evaluating of the  $M$  eigenfunctions  $\psi_{1},\ldots ,\psi_{M}$  on the input  $\mathbf{X}$ .
The second source of randomness in the KRR's test error comes from the empirical evaluation of the dot product of the eigenfunction  $\psi_{k}$ 's and the orthogonal complement  $\psi_{>M}$ :
Definition B.4 (Error Vector).  $E \stackrel{\mathrm{def.}}{=} \frac{1}{N} \Psi \psi_{>M}(\mathbf{X})$  is called the error vector.
The random matrix  $\pmb{\Delta}$  and the random vector  $\pmb{E}$  are centered; i.e.  $\mathbb{E}_{\mathbf{X}}[\pmb{\Delta}] = 0$  and  $\mathbb{E}_{\mathbf{X}}[\pmb{E}] = 0$ .
B.3 Bias-Variance Decomposition
The derivation of several contemporary KRR generalization bounds [6, 26, 27] involves the classical Bias-Variance Trade-off:
Proposition B.5 (Bias-Variance Trade-off). Fix a sample  $\mathbf{Z}$ . Recall the definition 3.4 of test error  $\mathcal{R}_{\mathbf{Z},\lambda}$ , bias, and variance:
$$
\mathcal {R} _ {\mathbf {Z}, \lambda} \stackrel {\text {d e f .}} {=} \mathbb {E} _ {x, \epsilon} \left[ (f _ {\mathbf {Z}, \lambda} (x) - \tilde {f} (x)) ^ {2} \right] = \mathbb {E} _ {\epsilon} \left[ \int_ {\mathcal {X}} \left(f _ {\mathbf {Z}, \lambda} (x) - \tilde {f} (x)\right) ^ {2} d \rho (x) \right];
$$
$$
b i a s \stackrel {\mathrm {d e f .}} {=} \int_ {\mathcal {X}} \left(f _ {(\mathbf {X}, \tilde {f} (\mathbf {X})), \lambda} (x) - \tilde {f} (x)\right) ^ {2} d \rho (x);
$$
variance  $\stackrel{\mathrm{def.}}{=} \mathcal{R}_{\mathbf{Z},\lambda} - bias.$
Then, we can write  $\text{variance}_{test} = \mathbb{E}_{x,\varepsilon}\left(\mathbf{K}_x^\top \mathbf{R}\varepsilon\right)^2$  and hence the test error  $\mathcal{R}_{\mathbf{Z},\lambda}$  admits a decomposition:
$$
R _ {\mathbf {Z}, \lambda} = b i a s + \mathbb {E} _ {x, \varepsilon} \left(\mathbf {K} _ {x} ^ {\top} \mathbf {R} \boldsymbol {\varepsilon}\right) ^ {2}.
$$
Proof. See the proof of Theorem C.8.
In this section, we will derive the essential lemmata and propositions for proving the main theorems.
C.1 Formula Derivation
We begin with writing the test error in convenient forms.
C.1.1 Bias
We first derive, from the definition of the bias, a convenient expression to proceed:
Proposition C.1 (Bias Expression). Let  $\Psi_{>M} \stackrel{\mathrm{def.}}{=} [\psi_{>M}(x_i)]_{i=1}^N$  as an  $1 \times N$ -row vector;  $\binom{\Psi}{\Psi_{>M}}$  as an  $(M+1) \times N$  matrix. Denote  $\mathbf{P} \stackrel{\mathrm{def.}}{=} \left(\mathbf{P}_{\leq M} \quad \mathbf{P}_{>M}\right) = \boldsymbol{\Lambda} \boldsymbol{\Psi} \mathbf{R} \left(\boldsymbol{\Psi}^\top \quad \boldsymbol{\Psi}_{>M}^\top\right) \in \mathbb{R}^{M \times (M+1)}$ ,  $\mathbf{P}_{\leq M} \in \mathbb{R}^{M \times M}$  and  $\mathbf{P}_{>M} \in \mathbb{R}^{M \times 1}$ . Then the bias admits the following expression:
$$
\text {b i a s} = \underbrace {\tilde {\gamma} _ {> M} ^ {2}} _ {\text {F i n i t e R a n k E r r o r}} + \underbrace {\| \tilde {\gamma} - \mathbf {P} _ {\leq M} \tilde {\gamma} - \tilde {\gamma} _ {> M} \mathbf {P} _ {> M} \| _ {2} ^ {2}} _ {\text {F i t t i n g E r r o r}}.
$$
Proof. Recall that, by equations (18) and (19), we can write
$$
\tilde {f} (x) = \tilde {\gamma} ^ {\top} \boldsymbol {\psi} (x) + \tilde {\gamma} _ {> M} \psi_ {> M} (x),
$$
$$
f _ {(\mathbf {X}, \tilde {f} (\mathbf {X})), \lambda} (x) = (\tilde {\boldsymbol {\gamma}} ^ {\top} \boldsymbol {\Psi} + \tilde {\boldsymbol {\gamma}} _ {> M} \boldsymbol {\Psi} _ {> M} ^ {\top}) \mathbf {R} \boldsymbol {\Psi} ^ {\top} \boldsymbol {\Lambda} \boldsymbol {\Psi} (x).
$$
$$
\begin{array}{l} \operatorname {b i a s} = \mathbb {E} _ {x} \left[ \left(\tilde {\boldsymbol {\gamma}} ^ {\top} \boldsymbol {\psi} (x) + \tilde {\gamma} _ {> M} \boldsymbol {\psi} _ {> M} (x) - (\tilde {\boldsymbol {\gamma}} ^ {\top} \boldsymbol {\Psi} + \tilde {\gamma} _ {> M} \boldsymbol {\Psi} _ {> M} ^ {\top}) \mathbf {R} \boldsymbol {\Psi} ^ {\top} \boldsymbol {\Lambda} \boldsymbol {\Psi} (x)\right) ^ {2} \right] \\ = \left\| \binom {\tilde {\gamma}} {\tilde {\gamma} _ {> M}} - \binom {\mathbf {P} \left(\tilde {\gamma} _ {\tilde {\gamma} > M}\right)} {0} \right\| _ {2} ^ {2} \tag {20} \\ = \underbrace {\tilde {\gamma} _ {> M} ^ {2}} _ {\text {F i n i t e R a n k E r r o r}} + \underbrace {\| \tilde {\gamma} - \mathbf {P} _ {\leq M} \tilde {\gamma} - \tilde {\gamma} _ {> M} \mathbf {P} _ {> M} \| _ {2} ^ {2}} _ {\text {F i t t i n g E r r o r}}, \\ \end{array}
$$
in line (20), we use Parseval's identity.
We proceed by reformulating the projection matrix  $\mathbf{P}$ , first with the left matrix  $\mathbf{P}_{\leq M}$ :
Lemma C.2. Recall the following notations
$$
\mathbf {K} \stackrel {{\text {d e f .}}} {{=}} \boldsymbol {\Psi} ^ {\top} \boldsymbol {\Lambda} \boldsymbol {\Psi}, \quad \mathbf {R} \stackrel {{\text {d e f .}}} {{=}} \left(\mathbf {K} + \lambda N \mathbf {I} _ {M}\right) ^ {- 1}, \quad \mathbf {P} _ {\leq M} \stackrel {{\text {d e f .}}} {{=}} \boldsymbol {\Lambda} \boldsymbol {\Psi} \mathbf {R} \boldsymbol {\Psi} ^ {\top}.
$$
Define the symmetric random matrix  $\mathbf{B} \stackrel{\mathrm{def.}}{=} (\mathbf{I}_M + \pmb{\Delta} + \lambda \pmb{\Lambda}^{-1})^{-1}$ . It holds that
$$
\mathbf {P} _ {\leq M} = \mathbf {I} _ {M} - \lambda \mathbf {B} \boldsymbol {\Lambda} ^ {- 1}.
$$
Proof. We first observe that
$$
\begin{array}{l} \Psi \Psi^ {\top} \mathbf {P} _ {\leq M} = \Psi \Psi^ {\top} \boldsymbol {\Lambda} \Psi \mathbf {R} \Psi^ {\top} (21) \\ = \boldsymbol {\Psi} \mathbf {K} (\mathbf {K} + \lambda N \mathbf {I} _ {M}) ^ {- 1} \boldsymbol {\Psi} ^ {\top} \\ = \boldsymbol {\Psi} \left(\mathbf {I} _ {M} - \lambda N (\mathbf {K} + \lambda N \mathbf {I} _ {M}) ^ {- 1}\right) \boldsymbol {\Psi} ^ {\top} \\ = \boldsymbol {\Psi} \boldsymbol {\Psi} ^ {\top} - \lambda N \boldsymbol {\Psi} (\mathbf {K} + \lambda N \mathbf {I} _ {M}) ^ {- 1} \boldsymbol {\Psi} ^ {\top}. (22) \\ \end{array}
$$
From lines (21)-(22) and the definition of the fluctuation matrix  $\Delta$  we deduce
$$
\frac {1}{N} \boldsymbol {\Psi} \boldsymbol {\Psi} ^ {\top} (\mathbf {I} _ {M} - \mathbf {P} _ {\leq M}) = \lambda \boldsymbol {\Psi} (\mathbf {K} + \lambda N \mathbf {I} _ {M}) ^ {- 1} \boldsymbol {\Psi} ^ {\top}
$$
$$
\left(\mathbf {I} _ {M} + \boldsymbol {\Delta}\right) \left(\mathbf {I} _ {M} - \mathbf {P} _ {\leq M}\right) = \lambda \boldsymbol {\Psi} \mathbf {R} \boldsymbol {\Psi} ^ {\top}
$$
$$
\left(\mathbf {I} _ {M} + \boldsymbol {\Delta}\right) \left(\mathbf {I} _ {M} - \mathbf {P} _ {\leq M}\right) = \lambda \boldsymbol {\Lambda} ^ {- 1} \mathbf {P} _ {\leq M}
$$
$$
(\boldsymbol {\Lambda} + \boldsymbol {\Lambda} \boldsymbol {\Delta}) (\mathbf {I} _ {M} - \mathbf {P} _ {\leq M}) = \lambda \mathbf {P} _ {\leq M}
$$
$$
\boldsymbol {\Lambda} + \boldsymbol {\Lambda} \boldsymbol {\Delta} = (\boldsymbol {\Lambda} + \boldsymbol {\Lambda} \boldsymbol {\Delta} + \lambda \mathbf {I} _ {M}) \mathbf {P} _ {\leq M}. \tag {23}
$$
Rearranging (23) and applying the definition of  $B$  we find that
$$
\begin{array}{l} \mathbf {P} _ {\leq M} = \left(\boldsymbol {\Lambda} + \boldsymbol {\Lambda} \boldsymbol {\Delta} + \lambda \mathbf {I} _ {M}\right) ^ {- 1} (\boldsymbol {\Lambda} + \boldsymbol {\Lambda} \boldsymbol {\Delta}) (24) \\ = \mathbf {I} _ {M} - \lambda (\boldsymbol {\Lambda} + \boldsymbol {\Lambda} \boldsymbol {\Delta} + \lambda \mathbf {I} _ {M}) ^ {- 1} \\ = \mathbf {I} _ {M} - \lambda (\boldsymbol {\Lambda} + \boldsymbol {\Lambda} \boldsymbol {\Delta} + \lambda \mathbf {I} _ {M}) ^ {- 1} \boldsymbol {\Lambda} \boldsymbol {\Lambda} ^ {- 1} \\ = \mathbf {I} _ {M} - \lambda \mathbf {B} \boldsymbol {\Lambda} ^ {- 1}. (25) \\ \end{array}
$$
Arguing analogously for the right matrix  $\mathbf{P}_{>M}$ , we draw the subsequent similar conclusion.
Lemma C.3. Recall the following notations
$$
\mathbf {K} \stackrel {{\text {d e f .}}} {{=}} \boldsymbol {\Psi} ^ {\top} \boldsymbol {\Lambda} \boldsymbol {\Psi}, \quad \mathbf {R} \stackrel {{\text {d e f .}}} {{=}} \left(\mathbf {K} + \lambda N \mathbf {I} _ {M}\right) ^ {- 1}, \quad \mathbf {P} _ {> M} \stackrel {{\text {d e f .}}} {{=}} \boldsymbol {\Lambda} \boldsymbol {\Psi} \mathbf {R} \boldsymbol {\Psi} _ {> M} ^ {\top},
$$
$$
\boldsymbol {E} \stackrel {{\text {d e f .}}} {{=}} \frac {1}{N} \boldsymbol {\Psi} \boldsymbol {\Psi} _ {> M} ^ {\top}, \quad \mathbf {B} \stackrel {{\text {d e f .}}} {{=}} (\mathbf {I} _ {M} + \boldsymbol {\Delta} + \lambda \boldsymbol {\Lambda} ^ {- 1}) ^ {- 1}.
$$
We have that  $\mathbf{P}_{>M} = \mathbf{B}\mathbf{E}$
Proof. Similarly to (21)-(22) we note that