SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Finally, note that from Lemma C.1, for any  $\lambda$ , there exists  $\psi$  such that  $v(0;\phi,\psi) = v(-\lambda;\phi,\phi)$ . This implies that  $\mathcal{R}(0;\phi,\psi) = \mathcal{R}(\lambda;\phi,\phi)$ . Then we have  $\min_{\psi \geq \phi}\mathcal{R}(0;\phi,\psi) \leq \min_{\lambda \geq 0}\mathcal{R}(\lambda;\phi,\phi)$ . Conversely, since for any  $\psi$ , there exists  $\lambda$  such that  $v(0;\phi,\psi) = v(-\lambda;\phi,\phi)$ , we also have  $\min_{\psi \geq \phi}\mathcal{R}(0;\phi,\psi) \geq \min_{\lambda \geq 0}\mathcal{R}(\lambda;\phi,\phi)$ . Combining the two inequalities yields the conclusion.
C.2 Technical lemmas
In this section, we gather results on certain analytic properties of the fixed-point solution  $v(-\lambda; \phi)$  defined in (40).
Lemma C.1 (Contour of fixed-point solutions). As  $n, p \to \infty$  such that  $p / n \to \phi \in (0, \infty)$ , for any  $\bar{\psi} \in [\phi, +\infty]$ , there exists a unique value  $\bar{\lambda} \geq 0$  (or conversely for  $\bar{\lambda} \in [0, \infty]$ , there exists a unique value  $\bar{\psi} \in [\phi \vee 1, \infty]$ ) such that for all  $(\lambda, \psi)$  on the path
$$
\mathcal {P} = \left\{\left(1 - \theta\right) \cdot (\bar {\lambda}, \phi) + \theta \cdot \left(0, \bar {\psi}\right) \mid \theta \in [ 0, 1 ] \right\},
$$
$$
v _ {p} (- \lambda ; \psi) = v (- \bar {\lambda}; \phi) = v (- 0; \bar {\psi}).
$$
where  $v_{p}(-\lambda ;\psi)$  is as defined in (40).
Proof. Since when  $\phi < 1$ ,  $v(0; \psi) = +\infty$  for all  $\psi \in [\phi, 1]$ , we can restrict ourselves to  $\psi \in [\phi \vee 1, +\infty]$ . From Lemma E.11 (1) of [13], the function  $\psi \mapsto v(0; \psi)$  is strictly decreasing over  $\psi \in [\phi \vee 1, \infty]$  with range
$$
v (0; \phi \lor 1) = \left\{ \begin{array}{l l} v (0; \phi), & \phi \in (1, \infty) \\ \lim  _ {\psi \to 1 ^ {+}} v (0; \psi) = + \infty , & \phi \in (0, 1 ] \end{array} \right., \qquad v (0; + \infty) := \lim  _ {\psi \to + \infty} v (0; \psi) = 0.
$$
From Lemma E.12 (3) of [13], the function  $\lambda \mapsto v(-\lambda; \phi)$  is strictly decreasing over  $\lambda \in [0, \infty]$  with range
$$
v (0; \phi) = \left\{ \begin{array}{l l} v (0; \phi), & \phi \in (1, \infty) \\ \lim  _ {\lambda \to 0 ^ {+}} v (- \lambda ; \phi) = + \infty , & \phi \in (0, 1 ] \end{array} \right., \qquad v (- \infty ; \phi) := \lim  _ {\lambda \to + \infty} v (- \lambda ; \phi) = 0.
$$
Note that  $v(0; \phi \lor 1) = v(0; \phi)$ . For  $\overline{\psi} \in [\phi \lor 1, \infty]$ , by the intermediate value theorem, there exists a unique  $\bar{\lambda} \in [0, \infty]$  such that  $v(-\bar{\lambda}; \phi) = v(0; \bar{\psi})$ . Furthermore, when  $\bar{\psi} \leq 1$ ,  $\bar{\lambda} = 0$  is also the unique
value such that  $v(0; \overline{\psi}) = v(-\overline{\lambda}; \phi)$ . Conversely, for  $\overline{\lambda} \in [0, \infty]$ , there also exists  $\overline{\psi} \in [\phi \vee 1, \infty]$  such that  $v(-\overline{\lambda}; \phi) = v(0; \overline{\psi})$ .
Based on the definition of fixed-point solutions, it follows that
$$
\frac {1}{v (- \bar {\lambda} ; \phi)} = \bar {\lambda} + \phi \int \frac {r}{1 + v (- \bar {\lambda} ; \phi) r} d H _ {p} (r) = \bar {\psi} \int \frac {r}{1 + v (0 ; \bar {\psi}) r} d H _ {p} (r) = \frac {1}{v (0 ; \bar {\psi})}.
$$
Then, for any  $(\lambda ,\psi) = (1 - \theta)(\bar{\lambda},\phi) + \theta (0,\bar{\psi})$  on the path  $\mathcal{P}$ , we have
$$
\begin{array}{l} \frac {1}{v (- \bar {\lambda} ; \phi)} = (1 - \theta) \frac {1}{v (- \bar {\lambda} ; \phi)} + \theta \frac {1}{v (0 ; \bar {\psi})} \\ = (1 - \theta) \bar {\lambda} + (1 - \theta) \phi \int \frac {r}{1 + v (- \bar {\lambda} ; \phi) r} d H _ {p} (r) + \theta \bar {\psi} \int \frac {r}{1 + v (0 ; \bar {\psi}) r} d H _ {p} (r) \\ = \lambda + \psi \int \frac {r}{1 + v (- \bar {\lambda} ; \phi) r} d H _ {p} (r). \\ \end{array}
$$
Because  $v_{p}(-\lambda ;\psi)$  is the unique solution to the fixed-point equation:
$$
\frac {1}{v _ {p} (- \lambda ; \psi)} = \lambda + \psi \int \frac {r}{1 + v _ {p} (- \lambda ; \psi)} \mathrm {d} H _ {p} (r),
$$
it then follows that  $v_{p}(-\lambda ;\psi) = v(-\bar{\lambda};\phi) = v(0;\bar{\psi})$ .
D Asymptotic equivalents, concentration results, and other useful lemmas
D.1 Full-ensemble resolvents
Lemma D.1 (Full-ensemble resolvents). Let  $\widehat{\pmb{\Sigma}} = \pmb{X}^{\top}\pmb{X} / n$ ,  $\widehat{\pmb{\Sigma}}_j = \pmb{X}^{\top}\pmb{L}_{I_j}\pmb{X} / k$  where  $I_{j}\sim \mathcal{I}_{k}$ . Let  $\pmb{\Sigma}_{1\cap 2} = \pmb{X}^{\top}\pmb{L}_{I_1\cap I_2}\pmb {X} / |I_1\cap I_2|$  and  $\pmb {C}\in \mathbb{R}^{p\times p}$  with bounded operator norm almost surely. As  $n,p,k\to \infty$ ,  $p / n\rightarrow \phi$ ,  $p / k\rightarrow \psi$ , the following asymptotic equivalences hold:
(1) Basic ridge resolvent:
$$
\lambda \mathbb {E} _ {I _ {1} \sim \mathcal {I} _ {k}} [ (\widehat {\pmb {\Sigma}} _ {1} + \lambda \pmb {I} _ {p}) ^ {- 1} ] \simeq (v _ {p} (- \lambda ; \psi) \pmb {\Sigma} + \pmb {I} _ {p}) ^ {- 1}.
$$
(2) Bias resolvent with  $C \perp X$ :
$$
\begin{array}{l} \mathbb {E} _ {I _ {1}, I _ {2} \sim \mathcal {I} _ {k}} [ \lambda^ {2} (\widehat {\pmb {\Sigma}} _ {1} + \lambda \pmb {I} _ {p}) ^ {- 1} \pmb {C} (\widehat {\pmb {\Sigma}} _ {2} + \lambda \pmb {I} _ {p}) ^ {- 1} ] \\ \simeq \left(v _ {p} (- \lambda ; \psi) \pmb {\Sigma} + \pmb {I} _ {p}\right) ^ {- 1} \left(\widetilde {v} _ {p} (- \lambda ; \phi , \psi , \pmb {C}) \pmb {\Sigma} + \pmb {C}\right) \left(v _ {p} (- \lambda ; \psi) \pmb {\Sigma} + \pmb {I} _ {p}\right) ^ {- 1}. \\ \end{array}
$$
(3) Variance resolvent with  $C \perp X$ :
$$
\begin{array}{l} \mathbb {E} _ {I _ {1}, I _ {2} \sim \mathcal {I} _ {k}} \left[ (\widehat {\boldsymbol {\Sigma}} _ {1} + \lambda \boldsymbol {I} _ {p}) ^ {- 1} \widehat {\boldsymbol {\Sigma}} _ {1 \cap 2} (\widehat {\boldsymbol {\Sigma}} _ {2} + \lambda \boldsymbol {I} _ {p}) ^ {- 1} \boldsymbol {C} \right] \\ \simeq \phi^ {- 1} \widetilde {v} _ {v} (- \lambda ; \phi , \psi) (v _ {p} (- \lambda ; \psi) \boldsymbol {\Sigma} + \boldsymbol {I} _ {p}) ^ {- 2} \boldsymbol {\Sigma} \boldsymbol {C}. \\ \end{array}
$$
(4) Bias resolvent with  $C = \widehat{\Sigma}$ :
$$
\begin{array}{l} \mathbb {E} _ {I _ {1}, I _ {2} \sim \mathcal {I} _ {k}} \left[ \lambda^ {2} (\widehat {\boldsymbol {\Sigma}} _ {1} + \lambda \boldsymbol {I} _ {p}) ^ {- 1} \widehat {\boldsymbol {\Sigma}} (\widehat {\boldsymbol {\Sigma}} _ {2} + \lambda \boldsymbol {I} _ {p}) ^ {- 1} \right] \\ \simeq \widetilde {d} (- \lambda ; \phi , \psi) (1 + \widetilde {v} _ {p} (- \lambda ; \phi , \psi)) (v _ {p} (- \lambda ; \psi) \pmb {\Sigma} + \pmb {I} _ {p}) ^ {- 2} \pmb {\Sigma}. \\ \end{array}
$$
Here  $v(-\lambda ;\phi)$  is as defined in (41), and we let  $\widetilde{v}_p(-\lambda ;\phi ,\psi) = \widetilde{v}_p(-\lambda ;\phi ,\psi ,\Sigma)$
$$
\widetilde {v} _ {p} (- \lambda ; \phi , \psi , \boldsymbol {C}) = \frac {\underset {k , n , p} {\lim} \phi \mathrm {t r} [ \boldsymbol {C} \boldsymbol {\Sigma} (v _ {p} (- \lambda ; \psi) \boldsymbol {\Sigma} + \boldsymbol {I} _ {p}) ^ {- 2} ] / p}{v _ {p} (- \lambda ; \psi) ^ {- 2} - \phi \int \frac {r ^ {2}}{(1 + v _ {p} (- \lambda ; \psi) r) ^ {2}} \mathrm {d} H _ {p} (r)},
$$
$$
\widetilde {v} _ {v} (- \lambda ; \phi , \psi) = \frac {1}{v _ {p} (- \lambda ; \psi) ^ {- 2} - \phi \int \frac {r ^ {2}}{(1 + v _ {p} (- \lambda ; \psi) r) ^ {2}} \mathrm {d} H _ {p} (r)},
$$
$$
\widetilde {d} (- \lambda ; \phi , \psi) = \left(1 - \phi \int \frac {v _ {p} (- \lambda ; \psi) r}{1 + v _ {p} (- \lambda ; \psi) r} \mathrm {d} H _ {p} (r)\right) ^ {2}.
$$
The empirical distribution  $H_{n}$  of eigenvalues can be replaced by the limiting distribution  $H$  whenever it exists.
Proof. The proofs for different parts is separated below.
(1) Basic ridge resolvent.
From Definition E.2, we know that  $\lambda (\widehat{\pmb{\Sigma}}_j + \lambda \pmb {I}_p)^{-1}\simeq (v_p(-\lambda ;\psi)\pmb {\Sigma} + \pmb {I}_p)^{-1}$ . By the definition of deterministic equivalents in Definition E.1, for any  $\pmb{A}\in \mathbb{R}^{p\times p}$  that has bounded trace norm and is independent to  $\pmb{X}$ , we have
$$
\mathrm {t r} [ \pmb {A} (\lambda (\widehat {\pmb {\Sigma}} _ {j} + \lambda \pmb {I} _ {p}) ^ {- 1} - (v _ {p} (- \lambda ; \psi) \pmb {\Sigma} + \pmb {I} _ {p}) ^ {- 1}) ] \stackrel {\mathrm {a . s .}} {\longrightarrow} 0.
$$
$$
\begin{array}{l} \mathrm {t r} [ \pmb {A} (\lambda \mathbb {E} _ {I _ {1} \sim \mathcal {I} _ {k}} [ (\widehat {\pmb {\Sigma}} _ {j} + \lambda \pmb {I} _ {p}) ^ {- 1} ] - (v _ {p} (- \lambda ; \psi) \pmb {\Sigma} + \pmb {I} _ {p}) ^ {- 1}) ] \\ = \mathbb {E} _ {I _ {1} \sim \mathcal {I} _ {k}} [ \operatorname {t r} \left[ \boldsymbol {A} \left(\lambda \left(\widehat {\boldsymbol {\Sigma}} _ {j} + \lambda \boldsymbol {I} _ {p}\right) ^ {- 1} - \left(v _ {p} (- \lambda ; \psi) \boldsymbol {\Sigma} + \boldsymbol {I} _ {p}\right) ^ {- 1}\right) \right] ] \xrightarrow {\mathrm {a . s .}} 0, \tag {28} \\ \end{array}
$$
(2) Bias resolvent with  $C \perp \perp X$ .
Denote  $M_{j} = (\widehat{\Sigma}_{j} + \lambda I_{p})^{-1}$  for  $j = 1,2$ . From Part (c) of the proof for Lemma S.2.4 in [12], it follows that for  $I_1, I_2 \sim \mathcal{I}_k$ ,
$$
\lambda^ {2} M _ {1} C M _ {2} \simeq (v _ {p} (- \lambda ; \psi) \pmb {\Sigma} + \pmb {I} _ {p}) ^ {- 1} (\widetilde {v} _ {p} (- \lambda ; \phi , \psi , \pmb {C}) \pmb {\Sigma} + \pmb {C}) (v _ {p} (- \lambda ; \psi) \pmb {\Sigma} + \pmb {I} _ {p}) ^ {- 1}.
$$
By the same argument as in (28), the conclusion follows.
(3) Variance resolvent with  $C \perp X$ .
From Lemma E.8 (3) of [13], it follows that for  $I_1, I_2 \sim \mathcal{I}_k$ ,
$$
M _ {1} \widehat {\boldsymbol {\Sigma}} _ {1 \cap 2} M _ {2} \boldsymbol {C} \simeq \phi^ {- 1} \widetilde {v} _ {v} (- \lambda ; \phi , \psi) (v _ {p} (- \lambda ; \psi) \boldsymbol {\Sigma} + \boldsymbol {I} _ {p}) ^ {- 2} \boldsymbol {\Sigma} \boldsymbol {C}.
$$
(4) Bias resolvent with  $C = \widehat{\Sigma}$
We begin by decomposing the object inside expectation into two terms:
$$
\lambda^ {2} M _ {1} \hat {\Sigma} M _ {2} = \frac {\left| I _ {1} \cup I _ {2} \right|}{n} \lambda^ {2} M _ {1} \hat {\Sigma} _ {1 \cup 2} M _ {2} + \frac {n - \left| I _ {1} \cup I _ {2} \right|}{n} \lambda^ {2} M _ {1} \hat {\Sigma} _ {(1 \cup 2) ^ {c}} M _ {2} \tag {29}
$$
where  $\widehat{\pmb{\Sigma}}_{1\cup 2} = \pmb{X}^{\top}\pmb{L}_{I_1\cup I_2}\pmb {X} / |I_1\cup I_2|$  and  $\widehat{\pmb{\Sigma}}_{(1\cup 2)^c} = \pmb{X}^\top (\pmb {I} - \pmb{L}_{I_1\cup I_2})\pmb {X} / (n - |I_1\cup I_2|)$ . Next, we analyze each of them.
For the first term, from Lemma D.6 (3) of [13] we have
$$
\begin{array}{l} M _ {1} \widehat {\Sigma} _ {1 \cup 2} M _ {2} \simeq v _ {p} (- \lambda ; \psi) ^ {2} (1 + \widetilde {v} _ {p} (- \lambda ; \phi , \psi)) \\ \left(\frac {2 (\psi - \phi)}{2 \psi - \phi} \frac {1}{\lambda v _ {p} (- \lambda ; \psi)} + \frac {\phi}{2 \psi - \phi}\right) \left(v _ {p} (- \lambda ; \psi) \boldsymbol {\Sigma} + \boldsymbol {I} _ {p}\right) ^ {- 2} \boldsymbol {\Sigma} \tag {30} \\ \end{array}
$$
where  $\widetilde{v}_p(-\lambda ;\phi ,\psi) = \widetilde{v}_p(-\lambda ;\phi ,\psi ,\Sigma)$
For the second term, from Lemma E.8 (1) of [13] and the product rule of calculus in Lemma E.3 (3), we have
$$
\lambda^ {2} M _ {1} \widehat {\Sigma} _ {(1 \cup 2) ^ {c}} M _ {2} \simeq \lambda^ {2} M _ {1} \Sigma M _ {2}.
$$
From Part (2) and Lemma E.3 (1), it follows that
$$
\lambda^ {2} M _ {1} \widehat {\boldsymbol {\Sigma}} _ {(1 \cup 2) ^ {c}} M _ {2} \simeq \lambda^ {2} M _ {1} \boldsymbol {\Sigma} M _ {2} \simeq \left(1 + \widetilde {v} _ {p} (- \lambda ; \phi , \psi)\right) \left(v _ {p} (- \lambda ; \psi) \boldsymbol {\Sigma} + \boldsymbol {I} _ {p}\right) ^ {- 2} \boldsymbol {\Sigma}. \tag {31}
$$
Finally, for the coefficients of the two terms, from Lemma S.8.3 of [12], we have that
$$
\frac {\left| I _ {1} \cup I _ {2} \right|}{n} = \frac {\left| I _ {1} \right| + \left| I _ {2} \right| - \left| I _ {1} \cap I _ {2} \right|}{n} \xrightarrow {\text {a . s .}} \frac {\phi (2 \psi - \phi)}{\psi^ {2}}, \quad \frac {n - \left| I _ {1} \cup I _ {2} \right|}{n} \xrightarrow {\text {a . s .}} \frac {(\psi - \phi) ^ {2}}{\psi^ {2}}. \tag {32}
$$
Combining (29)-(32) yields that
$$
\begin{array}{l} \lambda^ {2} M _ {1} \widehat {\Sigma} M _ {2} \\ \simeq \frac {1}{\psi^ {2}} \left(2 (\psi - \phi) \phi \lambda v _ {p} (- \lambda ; \psi) + \phi^ {2} \lambda^ {2} v _ {p} (- \lambda ; \psi) ^ {2} + (\psi - \phi) ^ {2}\right) \\ (1 + \widetilde {v} _ {p} (- \lambda ; \phi , \psi)) (v _ {p} (- \lambda ; \psi) \boldsymbol {\Sigma} + \boldsymbol {I} _ {p}) ^ {- 2} \boldsymbol {\Sigma} \\ = \frac {(\phi \lambda v _ {p} (- \lambda ; \psi) + \psi - \phi) ^ {2}}{\psi^ {2}} (1 + \widetilde {v} _ {p} (- \lambda ; \phi , \psi)) (v _ {p} (- \lambda ; \psi) \pmb {\Sigma} + \pmb {I} _ {p}) ^ {- 2} \pmb {\Sigma} \\ = \left(1 - \phi \int \frac {v _ {p} (- \lambda ; \psi) r}{1 + v _ {p} (- \lambda ; \psi) r} \mathrm {d} H _ {p} (r)\right) ^ {2} (1 + \widetilde {v} _ {p} (- \lambda ; \phi , \psi)) (v _ {p} (- \lambda ; \psi) \boldsymbol {\Sigma} + \boldsymbol {I} _ {p}) ^ {- 2} \boldsymbol {\Sigma} \\ \end{array}
$$
where the last equality follows by substituting
$$
\lambda = v _ {p} (- \lambda ; \psi) ^ {- 1} - \psi \int r (1 + v _ {p} (- \lambda ; \psi) r) ^ {- 1} d H _ {p} (r)
$$
based on the fixed-point equation.
D.2 Convergence of random linear and quadratic forms
Lemma D.2 (Concentration of linear form with independent components and varying coefficients). Let  $\mathbf{z}_i \in \mathbb{R}^p$  for  $i = 1, \ldots, n$  be a sequence of random vectors with i.i.d. entries  $z_{ij}$ ,  $j = 1, \ldots, p$  such that for each  $i, j$ ,  $\mathbb{E}[z_{ij}] = 0$ ,  $\mathbb{E}[z_{ij}^2] = 1$ ,  $\mathbb{E}[|z_{ij}|^{4 + \alpha}] \leq M_\alpha$  for some  $\alpha > 0$  and constant  $M_\alpha < \infty$ . Let  $\mathbf{Z} = [\mathbf{z}_1, \ldots, \mathbf{z}_n]^\top \in \mathbb{R}^{n \times p}$  be the random matrix formed by concatenating  $\mathbf{z}_i$ 's. Let  $g: \mathbb{R}^p \to \mathbb{R}$  be any measurable function such that  $\mathbb{E}[g(\mathbf{z}_i)] = 0$  and  $\mathbb{E}[z_i g(\mathbf{z}_i)] = 0$  for  $i = 1, \ldots, n$ . Let  $\mathbf{a}_p \in \mathbb{R}^p$  be a sequence of random vectors independent of  $\mathbf{z}_p$  such that  $\lim \sup_p \| \mathbf{a}_p \|^2 / p \leq M_0$  almost surely for a constant  $M_0 < \infty$ . Let  $D$  be a positive semidefinite matrix such that  $\lim \sup \| D \|_{\mathrm{op}} \leq M_0$  almost surely as  $p \to \infty$  for some constant  $M_0 < \infty$ . Then, as  $n, p \to \infty$  such that  $p / n \to \phi \in (0, \infty)$ , we have
$$
\left| \frac {1}{n} \sum_ {i = 1} ^ {n} \left[ \boldsymbol {a} ^ {\top} \left(\frac {\boldsymbol {D} ^ {\frac {1}{2}} \boldsymbol {Z} ^ {\top} \boldsymbol {Z} \boldsymbol {D} ^ {\frac {1}{2}}}{n} + \lambda \boldsymbol {I} _ {p}\right) ^ {- 1} \boldsymbol {D} ^ {\frac {1}{2}} \boldsymbol {Z} ^ {\top} \right] _ {i} g (\boldsymbol {z} _ {i}) \right| \xrightarrow {\text {a . s .}} 0. \tag {33}
$$
Proof. We will use the standard leave-one-out trick to break the dependence between the  $i$ -th component multiplier in the summation in (33) and  $g(z_{i})$  for each  $i = 1, \dots, n$ . To that end, using the Woodbury matrix identity, first observe that
$$
\begin{array}{l} \left(\frac {\pmb {D} ^ {\frac {1}{2}} \pmb {Z} ^ {\top} \pmb {Z} \pmb {D} ^ {\frac {1}{2}}}{n} + \lambda \pmb {I} _ {p}\right) ^ {- 1} \\ = \left(\frac {\boldsymbol {D} ^ {\frac {1}{2}} \boldsymbol {Z} _ {- i} ^ {\top} \boldsymbol {Z} _ {- i} \boldsymbol {D} ^ {\frac {1}{2}}}{n} + \frac {\boldsymbol {D} ^ {\frac {1}{2}} \boldsymbol {z} _ {i} \boldsymbol {z} _ {i} ^ {\top} \boldsymbol {D} ^ {\frac {1}{2}}}{n} + \lambda \boldsymbol {I} _ {p}\right) ^ {- 1} \\ = \left(\frac {\boldsymbol {D} ^ {\frac {1}{2}} \boldsymbol {Z} _ {- i} ^ {\top} \boldsymbol {Z} _ {- i} \boldsymbol {D} ^ {\frac {1}{2}}}{n} + \lambda \boldsymbol {I} _ {p}\right) ^ {- 1} \\ \end{array}
$$
$$
- \frac {\left(\frac {\boldsymbol {D} ^ {\frac {1}{2}} \boldsymbol {Z} _ {- i} ^ {\top} \boldsymbol {Z} _ {- i} \boldsymbol {D} ^ {\frac {1}{2}}}{n} + \lambda \boldsymbol {I} _ {p}\right) ^ {- 1} \frac {\boldsymbol {D} ^ {\frac {1}{2}} \boldsymbol {z} _ {i} \boldsymbol {z} _ {i} ^ {\top} \boldsymbol {D} ^ {\frac {1}{2}}}{n} \left(\frac {\boldsymbol {D} ^ {\frac {1}{2}} \boldsymbol {Z} _ {- i} ^ {\top} \boldsymbol {Z} _ {- i} \boldsymbol {D} ^ {\frac {1}{2}}}{n} + \lambda \boldsymbol {I} _ {p}\right) ^ {- 1}}{1 + \frac {\boldsymbol {z} _ {i} ^ {\top} \boldsymbol {D} ^ {\frac {1}{2}} \left(\frac {\boldsymbol {D} ^ {\frac {1}{2}} \boldsymbol {Z} _ {- i} ^ {\top} \boldsymbol {Z} _ {- i} \boldsymbol {D} ^ {\frac {1}{2}}}{n} + \lambda \boldsymbol {I} _ {p}\right) ^ {- 1} \boldsymbol {D} ^ {\frac {1}{2}} \boldsymbol {z} _ {i}}{n}}. \tag {34}
$$
Plugging (34) back into (33), we expand the desired sum into:
$$
\begin{array}{l} \frac {1}{n} \sum_ {i = 1} ^ {n} \left[ \boldsymbol {a} ^ {\top} \left(\frac {\boldsymbol {D} ^ {\frac {1}{2}} \boldsymbol {Z} ^ {\top} \boldsymbol {Z} \boldsymbol {D} ^ {\frac {1}{2}}}{n} + \lambda \boldsymbol {I} _ {p}\right) ^ {- 1} \boldsymbol {D} ^ {\frac {1}{2}} \boldsymbol {Z} ^ {\top} \right] _ {i} g (\boldsymbol {z} _ {i}) \\ = \frac {1}{n} \sum_ {i = 1} ^ {n} \boldsymbol {a} ^ {\top} \left(\frac {\boldsymbol {D} ^ {\frac {1}{2}} \boldsymbol {Z} ^ {\top} \boldsymbol {Z} \boldsymbol {D} ^ {\frac {1}{2}}}{n} + \lambda \boldsymbol {I} _ {p}\right) ^ {- 1} \boldsymbol {D} ^ {\frac {1}{2}} \boldsymbol {z} _ {i} g (\boldsymbol {z} _ {i}) \\ = \frac {1}{n} \sum_ {i = 1} ^ {n} \boldsymbol {a} ^ {\top} \left(\frac {\boldsymbol {D} ^ {\frac {1}{2}} \boldsymbol {Z} _ {- i} ^ {\top} \boldsymbol {Z} _ {- i} \boldsymbol {D} ^ {\frac {1}{2}}}{n} + \lambda \boldsymbol {I} _ {p}\right) ^ {- 1} \boldsymbol {D} ^ {\frac {1}{2}} \boldsymbol {z} _ {i} g (\boldsymbol {z} _ {i}) \\ - \frac {1}{n} \sum_ {i = 1} ^ {n} \frac {\boldsymbol {a} ^ {\top} \left(\frac {\boldsymbol {D} ^ {\frac {1}{2}} \boldsymbol {Z} _ {- i} ^ {\top} \boldsymbol {Z} _ {- i} \boldsymbol {D} ^ {\frac {1}{2}}}{n} + \lambda \boldsymbol {I} _ {p}\right) ^ {- 1} \frac {\boldsymbol {D} ^ {\frac {1}{2}} \boldsymbol {z} _ {i} \boldsymbol {z} _ {i} ^ {\top} \boldsymbol {D} ^ {\frac {1}{2}}}{n} \left(\frac {\boldsymbol {D} ^ {\frac {1}{2}} \boldsymbol {Z} _ {- i} ^ {\top} \boldsymbol {Z} _ {- i} \boldsymbol {D} ^ {\frac {1}{2}}}{n} + \lambda \boldsymbol {I} _ {p}\right) ^ {- 1} \boldsymbol {D} ^ {\frac {1}{2}} \boldsymbol {z} _ {i} g (\boldsymbol {z} _ {i})}{1 + \frac {\boldsymbol {z} _ {i} ^ {\top} \boldsymbol {D} ^ {\frac {1}{2}} \left(\frac {\boldsymbol {D} ^ {\frac {1}{2}} \boldsymbol {Z} _ {- i} ^ {\top} \boldsymbol {Z} _ {- i} \boldsymbol {D} ^ {\frac {1}{2}}}{n} + \lambda \boldsymbol {I} _ {p}\right) ^ {- 1} \boldsymbol {D} ^ {\frac {1}{2}} \boldsymbol {z} i}{n}} \\ = \frac {1}{n} \sum_ {i = 1} ^ {n} \boldsymbol {b} _ {i} ^ {\top} \boldsymbol {z} _ {i} g (\boldsymbol {z} _ {i}) - \frac {1}{n} \sum_ {i = 1} ^ {n} \frac {\boldsymbol {b} _ {i} ^ {\top} d _ {i} \boldsymbol {z} _ {i} g (\boldsymbol {z} _ {i})}{1 + d _ {i}} (35) \\ \leq \frac {1}{n} \sum_ {i = 1} ^ {n} \left(1 + d _ {i}\right) \boldsymbol {b} _ {i} ^ {\top} \boldsymbol {z} _ {i} g \left(\boldsymbol {z} _ {i}\right), (36) \\ \end{array}
$$
where in step (35), we denote by:
$$
\boldsymbol {b} _ {i} = \boldsymbol {D} ^ {\frac {1}{2}} \left(\frac {\boldsymbol {D} ^ {\frac {1}{2}} \boldsymbol {Z} _ {- i} ^ {\top} \boldsymbol {Z} _ {- i} \boldsymbol {D} ^ {\frac {1}{2}}}{n} + \lambda \boldsymbol {I} _ {p}\right) ^ {- 1} \boldsymbol {a},
$$
$$
d _ {i} = \frac {\boldsymbol {z} _ {i} ^ {\top} \boldsymbol {D} ^ {\frac {1}{2}} \left(\frac {\boldsymbol {D} ^ {\frac {1}{2}} \boldsymbol {Z} _ {- i} ^ {\top} \boldsymbol {Z} _ {- i} \boldsymbol {D} ^ {\frac {1}{2}}}{n} + \lambda \boldsymbol {I} _ {p}\right) ^ {- 1} \boldsymbol {D} ^ {\frac {1}{2}} \boldsymbol {z} _ {i}}{n},
$$
and step (36) follows since  $d_{i} \geq 0$ . It is easy to check that  $\lim \sup_{p} \| b_{i} \|_{2}^{2} / p \leq C_{1} < \infty$ , and  $d_{i} \xrightarrow{\text{a.s.}} C_{2} < \infty$  for some constants  $C_{1}$  and  $C_{2}$ . Appealing to Lemma S.8.5 of [51], (36) almost surely converges to 0. This finishes the proof.
Lemma D.3 (Concentration of sum of quadratic forms with independent components and independent varying inner matrices). Let  $\mathbf{z}_p \in \mathbb{R}^p$  be a sequence of random vectors with i.i.d. entries  $z_{pi}$ ,  $i = 1, \ldots, p$  such that for each  $i$ ,  $\mathbb{E}[z_{pi}] = 0$ ,  $\mathbb{E}[z_{pi}^2] = 1$ ,  $\mathbb{E}[|z_{pi}|^{4 + \alpha}] \leq M_\alpha$  for some  $\alpha > 0$  and constant  $M_\alpha < \infty$ . Let  $\mathbf{Z} = [\mathbf{z}_1, \dots, \mathbf{z}_n]^\top \in \mathbb{R}^{n \times p}$  be the design matrix. Let  $g: \mathbb{R}^p \to \mathbb{R}$  be any measurable function such that  $\mathbb{E}[z_i g(\mathbf{z}_i)] = 0$  and  $\mathbb{E}[g(\mathbf{z}_i)] = 1$ . Let  $\mathbf{D}$  be a positive semidefinite matrix such that  $\lim \sup \| \mathbf{D} \|_{\mathrm{op}} \leq M_0$  almost surely as  $p \to \infty$  for some constant  $M_0 < \infty$ . Then, as  $n, p \to \infty$  such that  $p / n \to \phi \in (0, \infty)$ ,
$$
\left| \frac {1}{n} \sum_ {1 \leq i \neq j \leq n} \left(\frac {\boldsymbol {Z D Z} ^ {\top}}{n} + \lambda \boldsymbol {I} _ {n}\right) _ {i j} ^ {- 1} g \left(\boldsymbol {z} _ {i}\right) g \left(\boldsymbol {z} _ {j}\right) \right| \xrightarrow {\text {a . s .}} 0. \tag {37}
$$
Proof. The strategy in the proof is to express each of the  $ij$ -the entry of the resolvent in (37) such that the dependence on  $(z_i, z_j)$  and the rest of  $(z_k : k \neq i, j)$  is separated. One is then able to use the
uncorrelatedness of  $z$  and  $g(z)$  along with standard concentration of a quadratic form with respect to an independent matrix. Similar strategy has been used in [14] when analyzing kernel ridge regression under proportional asymptotics.
Denote  $\mathbf{Y} = (\mathbf{Z}\mathbf{D}\mathbf{Z}^{\top} / n + \lambda \mathbf{I}_n)^{-1}$ . For each pair of  $i,j\in [n]$  and  $i\neq j$ , we let  $\mathbf{Z}_{-(ij)}\in \mathbb{R}^{(n - 2)\times p}$  be the matrix comprising the  $n - 2$  rows of  $\mathbf{Z}$  excluding the  $i$ th and  $j$ th rows, and  $\mathbf{U}\in \mathbb{R}^{p\times 2}$  be the matrix with columns  $\mathbf{U}_{ij}\mathbf{e}_i = \mathbf{z}_i$ ,  $\mathbf{U}\mathbf{e}_j = \mathbf{z}_j$ . We finally define the matrices  $\mathbf{R}_{-(ij)}\in \mathbb{R}^{p\times p}$  as follows:
$$
\pmb {R} _ {- (i j)} = \lambda \pmb {D} ^ {1 / 2} \left(\pmb {D} ^ {1 / 2} \pmb {Z} _ {- (i j)} ^ {\top} \pmb {Z} _ {- (i j)} \pmb {D} ^ {1 / 2} / n + \lambda \mathbf {I} _ {p}\right) ^ {- 1} \pmb {D} ^ {1 / 2},
$$
and let  $\tilde{\mathbf{Y}}_{ij} = \{Y_{m,\ell}\}_{m,\ell \in \{i,j\}}$  be the submatrix of  $\mathbf{Y}$ . Then, using the block matrix inversion formula (see, e.g., Appendix A.1.4 of [45]) and the Woodbury matrix identity, one can show that
$$
\widetilde {\boldsymbol {Y}} _ {i j} = \left(\boldsymbol {U} _ {i j} ^ {\top} \boldsymbol {R} _ {- (i j)} \boldsymbol {U} _ {i j} / n + \lambda \mathbf {I} _ {2}\right) ^ {- 1},
$$
$$
Y _ {i j} = - \frac {\left\langle \boldsymbol {z} _ {i} , \boldsymbol {R} _ {- (i j)} \boldsymbol {z} _ {j} \right\rangle / n}{d _ {i j}},
$$
where  $d_{ij}$  is given by:
$$
d _ {i j} = \left(\lambda + \left\langle \boldsymbol {z} _ {i}, \boldsymbol {R} _ {- (i j)} \boldsymbol {z} _ {j} \right\rangle / n\right) \left(\lambda + \left\langle \boldsymbol {z} _ {i}, \boldsymbol {R} _ {- (i j)} \boldsymbol {z} _ {j} \right\rangle / n\right) - \left\langle \boldsymbol {z} _ {i}, \boldsymbol {R} _ {- (i j)} \boldsymbol {z} _ {j} \right\rangle^ {2} / n ^ {2}.
$$