SlowGuess/NeurIPS_2023_OCR huggingface dataset:
From (30), (32), (34), we find:
$$
\begin{array}{l} \langle G _ {i j} (z) \rangle = \lim  _ {n \rightarrow \infty} \int \left(\prod_ {k = 1} ^ {N + M} \prod_ {\tau = 1} ^ {n} d \eta_ {k} ^ {(\tau)}\right) \eta_ {i} ^ {(1)} \eta_ {j} ^ {(1)} \\ \times \exp \left\{- \frac {1}{2} \sum_ {\tau = 1} ^ {n} z \| \boldsymbol {\eta} ^ {(\tau)} \| ^ {2} - N \mathcal {Q} _ {\mu_ {Y}} ^ {(\alpha)} \left(\frac {\| \boldsymbol {X} \boldsymbol {a} ^ {(\tau)} \| ^ {2} \| \boldsymbol {b} ^ {(\tau)} \| ^ {2}}{N M}\right) - N \mathcal {Q} _ {\mu_ {W}} ^ {(\alpha)} \left(\frac {\| \boldsymbol {a} ^ {(\tau)} \| ^ {2} \| \boldsymbol {b} ^ {(\tau)} \| ^ {2}}{N M}\right) \right\} \tag {35} \\ \end{array}
$$
Now, we introduce delta functions  $\delta \big(p_1^{(\tau)} - \frac{\|a^{(\tau)}\|^2}{N}\big), \delta \big(p_2^{(\tau)} - \frac{\|b^{(\tau)}\|^2}{M}\big)$ , and  $\delta \big(p_3^{(\tau)} - \frac{\|Xa^{(\tau)}\|^2}{N}\big)$  and using them, the integral in (35) can be written as (for brevity we drop the limit term):
$$
\begin{array}{l} \langle G _ {i j} (z) \rangle = \int \big (\prod_ {k = 1} ^ {N + M} \prod_ {\tau = 1} ^ {n} d \eta_ {k} ^ {(\tau)} \big) \big (\prod_ {\tau = 1} ^ {n} d p _ {1} ^ {(\tau)} d p _ {2} ^ {(\tau)} d p _ {3} ^ {(\tau)} \big) \eta_ {i} ^ {(1)} \eta_ {j} ^ {(1)} \\ \times \prod_ {\tau = 1} ^ {n} \delta \left(p _ {1} ^ {(\tau)} - \frac {\| \boldsymbol {a} ^ {(\tau)} \| ^ {2}}{N}\right) \delta \left(p _ {2} ^ {(\tau)} - \frac {\| \boldsymbol {b} ^ {(\tau)} \| ^ {2}}{M}\right) \delta \left(p _ {3} ^ {(\tau)} - \frac {\| \boldsymbol {X} \boldsymbol {a} ^ {(\tau)} \| ^ {2}}{N}\right) \tag {36} \\ \times \exp \Big \{- \frac {1}{2} \sum_ {\tau = 1} ^ {n} z \| \pmb {\eta} ^ {(\tau)} \| ^ {2} - N \mathcal {Q} _ {\mu_ {Y}} ^ {(\alpha)} (p _ {2} ^ {(\tau)} p _ {3} ^ {(\tau)}) - N \mathcal {Q} _ {\mu_ {W}} ^ {(\alpha)} (p _ {1} ^ {(\tau)} p _ {2} ^ {(\tau)}) \Big \} \\ \end{array}
$$
In the next step, we replace each delta with its Fourier transform,  $\delta \left(p_1^\tau -\frac{1}{N}\| \pmb {a}^\tau \| ^2\right)\propto \int d\zeta_1^\tau \exp \Big\{-\frac{N}{2}\zeta_1^\tau \left(p_1^\tau -\frac{1}{N}\| \pmb {a}^\tau \| ^2\right)\Big\}$ . After rearranging, we find:
$$
\begin{array}{l} \langle G _ {i j} (z) \rangle \propto \int \big (\prod_ {\tau = 1} ^ {n} d p _ {1} ^ {(\tau)} d p _ {2} ^ {(\tau)} d p _ {3} ^ {(\tau)} d \zeta_ {1} ^ {(\tau)} d \zeta_ {2} ^ {(\tau)} d \zeta_ {3} ^ {(\tau)} \big) \\ \times \exp \left\{\frac {N}{2} \sum_ {\tau = 1} ^ {n} \mathcal {Q} _ {\mu_ {Y}} ^ {(\alpha)} (p _ {2} ^ {(\tau)} p _ {3} ^ {(\tau)}) + \mathcal {Q} _ {\mu_ {W}} ^ {(\alpha)} (p _ {1} ^ {(\tau)} p _ {2} ^ {(\tau)}) - \zeta_ {1} ^ {(\tau)} p _ {1} ^ {(\tau)} - \frac {1}{\alpha} \zeta_ {2} ^ {(\tau)} p _ {2} ^ {(\tau)} - \zeta_ {3} ^ {(\tau)} p _ {3} ^ {(\tau)} \right\} \\ \times \int \left(\prod_ {k = 1} ^ {N + M} \prod_ {\tau = 1} ^ {n} d \eta_ {k} ^ {(\tau)}\right) \eta_ {i} ^ {(1)} \eta_ {j} ^ {(1)} \\ \times \exp \left\{- \frac {1}{2} \sum_ {\tau = 1} ^ {n} z \| \boldsymbol {\eta} ^ {(\tau)} \| ^ {2} - \zeta_ {1} ^ {(\tau)} \| \boldsymbol {a} ^ {(\tau)} \| ^ {2} - \zeta_ {2} ^ {(\tau)} \| \boldsymbol {b} ^ {(\tau)} \| ^ {2} - \zeta_ {3} ^ {(\tau)} \| \boldsymbol {X} \boldsymbol {a} ^ {(\tau)} \| ^ {2} \right\} \tag {37} \\ \end{array}
$$
The inner integral in (37) is a Gaussian integral, and can be written as:
$$
\begin{array}{l} \int \left(\prod_ {k = 1} ^ {N + M} \prod_ {\tau = 1} ^ {n} d \eta_ {k} ^ {(\tau)}\right) \eta_ {i} ^ {(1)} \eta_ {j} ^ {(1)} \\ \times \exp \left\{\sum_ {\tau = 1} ^ {n} - \frac {1}{2} \boldsymbol {\eta} ^ {(\tau) \intercal} \left[ \begin{array}{c c} (z - \zeta_ {1} ^ {(\tau)}) \boldsymbol {I} _ {N} - \zeta_ {3} ^ {(\tau)} \boldsymbol {X} ^ {2} & \mathbf {0} \\ \mathbf {0} & (z - \zeta_ {2} ^ {(\tau)}) \boldsymbol {I} _ {M} \end{array} \right] \boldsymbol {\eta} ^ {(\tau)} \right\} \tag {38} \\ \end{array}
$$
Denote the matrix in the exponent by  $C_X^{(\tau)}$ . Its determinant reads:
$$
\det  C _ {X} ^ {(\tau)} = (z - \zeta_ {2} ^ {(\tau)}) ^ {M} \prod_ {k = 1} ^ {N} (z - \zeta_ {1} ^ {(\tau)} - \zeta_ {3} ^ {(\tau)} \lambda_ {k} ^ {2})
$$
where  $\lambda_{k}$ 's are eigenvalues of  $X$ . So replacing the formula for the Gaussian integrals, (37) can be written as:
$$
\begin{array}{l} \langle G _ {i j} (z) \rangle \propto \int \big (\prod_ {\tau = 1} ^ {n} d p _ {1} ^ {(\tau)} d p _ {2} ^ {(\tau)} d p _ {3} ^ {(\tau)} d \zeta_ {1} ^ {(\tau)} d \zeta_ {2} ^ {(\tau)} d \zeta_ {3} ^ {(\tau)} \big) \Big (C _ {X} ^ {(1)} ^ {- 1} \Big) _ {i j} \\ \times \exp \left\{- \frac {N n}{2} F _ {0} ^ {X} \left(\boldsymbol {p} _ {1}, \boldsymbol {p} _ {2}, \boldsymbol {p} _ {3}, \zeta_ {1}, \zeta_ {2}, \zeta_ {3}\right) \right\} \tag {39} \\ \end{array}
$$
$$
\begin{array}{l} F _ {0} ^ {X} (\boldsymbol {p} _ {1}, \boldsymbol {p} _ {2}, \boldsymbol {p} _ {3}, \boldsymbol {\zeta} _ {1}, \boldsymbol {\zeta} _ {2}, \boldsymbol {\zeta} _ {3}) = \frac {1}{n} \sum_ {\tau = 1} ^ {n} \left[ \frac {1}{N} \sum_ {k = 1} ^ {N} \ln (z - \zeta_ {1} ^ {(\tau)} - \zeta_ {3} ^ {(\tau)} \lambda_ {k} ^ {2}) + \frac {1}{\alpha} \ln (z - \zeta_ {2} ^ {(\tau)}) \right. \\ \left. - \mathcal {Q} _ {\mu_ {Y}} ^ {(\alpha)} \left(p _ {2} ^ {(\tau)} p _ {3} ^ {(\tau)}\right) - \mathcal {Q} _ {\mu_ {W}} ^ {(\alpha)} \left(p _ {1} ^ {(\tau)} p _ {2} ^ {(\tau)}\right) + \zeta_ {1} ^ {(\tau)} p _ {1} ^ {(\tau)} + \frac {1}{\alpha} \zeta_ {2} ^ {(\tau)} p _ {2} ^ {(\tau)} + \zeta_ {3} ^ {(\tau)} p _ {3} ^ {(\tau)} \right] \tag {40} \\ \end{array}
$$
In the large  $N$  limit, the integral in (39) can be computed using the saddle-points of the function  $F_0^X$ . In the evaluation of this integral, we use the replica symmetric ansatz that assumes a saddle-point of the form:
$$
\forall \tau \in \{1, \dots , n \}: \quad \left\{ \begin{array}{l l} p _ {1} ^ {\tau} = p _ {1}, & p _ {2} ^ {\tau} = p _ {2}, \quad p _ {3} ^ {\tau} = p _ {3} \\ \zeta_ {1} ^ {\tau} = \zeta_ {1}, & \zeta_ {2} ^ {\tau} = \zeta_ {2}, \quad \zeta_ {3} ^ {\tau} = \zeta_ {3} \end{array} \right.
$$
The saddle point is a solution of the set of equations:
$$
\left\{ \begin{array}{l} \zeta_ {1} ^ {*} = \frac {\mathcal {C} _ {\mu_ {W}} ^ {(\alpha)} \left(p _ {1} ^ {*} p _ {2} ^ {*}\right)}{p _ {1} ^ {*}}, \quad \zeta_ {2} ^ {*} = \frac {\alpha}{p _ {2} ^ {*}} \left(\mathcal {C} _ {\mu_ {W}} ^ {(\alpha)} \left(p _ {1} ^ {*} p _ {2} ^ {*}\right) + \mathcal {C} _ {\mu_ {Y}} ^ {(\alpha)} \left(p _ {2} ^ {*} p _ {3} ^ {*}\right)\right), \quad \zeta_ {3} ^ {*} = \frac {\mathcal {C} _ {\mu_ {Y}} ^ {(\alpha)} \left(p _ {2} ^ {*} p _ {3} ^ {*}\right)}{p _ {3} ^ {*}} \\ p _ {1} ^ {*} = \frac {1}{\zeta_ {3} ^ {*}} \mathcal {G} _ {\rho_ {X 2}} \left(\frac {z - \zeta_ {1} ^ {*}}{\zeta_ {3} ^ {*}}\right), \quad p _ {2} ^ {*} = \frac {1}{z - \zeta_ {2} ^ {*}}, \quad p _ {3} ^ {*} = \frac {z - \zeta_ {1} ^ {*}}{\zeta_ {3} ^ {* 2}} \mathcal {G} _ {\rho_ {X 2}} \left(\frac {z - \zeta_ {1} ^ {*}}{\zeta_ {3} ^ {*}}\right) - \frac {1}{\zeta_ {3} ^ {*}} \end{array} \right. \tag {41}
$$
Now, since the relation (39) and the solutions (41) hold for arbitrary indices  $i,j$ , we can state the relation in matrix form. The inverse of  $C_X^{*^{-1}}$ , and the block structure of  $\pmb{G}_{\mathcal{S}}(z)$  are computed in sections H.2. From (111), (112) we have (for sufficiently large  $N$ ):
$$
\begin{array}{l} \left\langle \boldsymbol {G} _ {\mathcal {S}} (z) \right\rangle_ {\boldsymbol {U} _ {1}, \boldsymbol {U} _ {2}, \boldsymbol {V} _ {1}, \boldsymbol {V} _ {2}} = \left\langle \left[ \begin{array}{c c} \frac {1}{z} \boldsymbol {I} _ {N} + \frac {1}{z} \boldsymbol {S} \boldsymbol {G} _ {S \tau S} \left(z ^ {2}\right) \boldsymbol {S} ^ {\intercal} & \boldsymbol {S} \boldsymbol {G} _ {S \tau S} \left(z ^ {2}\right) \\ \boldsymbol {G} _ {S \tau S} \left(z ^ {2}\right) \boldsymbol {S} ^ {\intercal} & z \boldsymbol {G} _ {S \tau S} \left(z ^ {2}\right) \end{array} \right] \right\rangle \tag {42} \\ = \left[ \begin{array}{c c} \frac {1}{\zeta_ {3} ^ {*}} \boldsymbol {G} _ {X ^ {2}} \big (\frac {z - \zeta_ {1} ^ {*}}{\zeta_ {3} ^ {*}} \big) & \boldsymbol {0} \\ \boldsymbol {0} & \frac {1}{z - \zeta_ {2} ^ {*}} \boldsymbol {I} _ {M} \end{array} \right] \\ \end{array}
$$
With this relation, we proceed to simplify the equations (41).
The normalized trace of the upper-left blocks of  $\left\langle G_{\mathcal{S}}(z)\right\rangle_{U_1,U_2,V_1,V_2}$  is:
$$
\begin{array}{l} \frac {1}{N} \sum_ {k = 1} ^ {N} \left[ \frac {1}{z} + \frac {1}{z} \frac {\gamma_ {k} ^ {2}}{z ^ {2} - \gamma_ {k} ^ {2}} \right] = \frac {1}{z} \frac {1}{N} \sum_ {k = 1} ^ {N} \left[ 1 + \frac {\gamma_ {k} ^ {2}}{z ^ {2} - \gamma_ {k} ^ {2}} \right] \\ = z \frac {1}{N} \sum_ {k = 1} ^ {N} \frac {1}{z ^ {2} - \gamma_ {k} ^ {2}} \tag {43} \\ = \frac {1}{2 N} \sum_ {k = 1} ^ {N} \left[ \frac {1}{z - \gamma_ {k}} + \frac {1}{z + \gamma_ {k}} \right] = \mathcal {G} _ {\bar {\mu} _ {S}} (z) \\ \end{array}
$$
and the normalized trace of the upper-left block in  $C_X^{* - 1}$  is  $\frac{1}{\zeta_3^*}\mathcal{G}_{\rho_{X^2}}\left(\frac{z - \zeta_1^*}{\zeta_3^*}\right) = p_1^*$ . Therefore, we have  $p_1^* = \mathcal{G}_{\bar{\mu}_S}(z)$ .
The normalized trace of lower-right block of  $\left\langle G_{S}(z)\right\rangle_{U_1,U_2,V_1,V_2}$  reads:
$$
\frac {1}{M} z \left[ \sum_ {k = 1} ^ {N} \frac {1}{z ^ {2} - \gamma_ {k} ^ {2}} + (M - N) \frac {1}{z ^ {2}} \right] = \frac {N}{M} \mathcal {G} _ {\bar {\mu} _ {S}} (z) + \frac {M - N}{M} \frac {1}{z} = \alpha \mathcal {G} _ {\bar {\mu} _ {S}} (z) + (1 - \alpha) \frac {1}{z} \tag {44}
$$
and the normalized trace of the lower-right block in  $C_X^{* - 1}$  is  $\frac{1}{z - \zeta_2^*} = p_2^*$ . Therefore, we have  $p_2^* = \alpha \mathcal{G}_{\bar{\mu}_S}(z) + (1 - \alpha)\frac{1}{z}$ . Moreover, we also have that  $\zeta_2^* = \alpha z\frac{z\mathcal{G}_{\bar{\mu}_S}(z) - 1}{\alpha z\mathcal{G}_{\bar{\mu}_S}(z) + 1 - \alpha}$ .
Therefore, the saddle point equations (41) can be rewritten in a simplified form, which does not involve  $\rho_{X^2}$ , as:
$$
\left\{ \begin{array}{l} \zeta_ {1} ^ {*} = \frac {\mathcal {C} _ {\mu W} ^ {(\alpha)} \left(p _ {1} ^ {*} p _ {2} ^ {*}\right)}{p _ {1} ^ {*}}, \quad \zeta_ {2} ^ {*} = \alpha z \frac {z \mathcal {G} _ {\bar {\mu} _ {S}} (z) - 1}{\alpha z \mathcal {G} _ {\bar {\mu} _ {S}} (z) + 1 - \alpha}, \quad \zeta_ {3} ^ {*} = \frac {\mathcal {C} _ {\mu Y} ^ {(\alpha)} \left(p _ {2} ^ {*} p _ {3} ^ {*}\right)}{p _ {3} ^ {*}} \\ p _ {1} ^ {*} = \mathcal {G} _ {\bar {\mu} _ {S}} (z), \quad p _ {2} ^ {*} = \alpha \mathcal {G} _ {\bar {\mu} _ {S}} (z) + (1 - \alpha) \frac {1}{z}, \quad p _ {3} ^ {*} = \frac {z - \zeta_ {1} ^ {*}}{\zeta_ {3} ^ {*}} \mathcal {G} _ {\bar {\mu} _ {S}} (z) - \frac {1}{\zeta_ {3} ^ {*}} \end{array} \right. \tag {45}
$$
Note that  $\zeta_1^*,\zeta_2^*$  can be computed from the observation matrix, and we only need to find  $\zeta_3^*$  satisfying the following equation:
$$
\left. \left(z - \zeta_ {1} ^ {*}\right) \mathcal {G} _ {\bar {\mu} _ {S}} (z) - 1 = \mathcal {C} _ {\mu_ {Y}} ^ {(\alpha)} \left(\frac {1}{\zeta_ {3} ^ {*}} \left[ \alpha \mathcal {G} _ {\bar {\mu} _ {S}} (z) + \frac {1 - \alpha}{z} \right] \left[ \left(z - \zeta_ {1} ^ {*}\right) \mathcal {G} _ {\bar {\mu} _ {S}} (z) - 1 \right]\right) \right. \tag {46}
$$
C.2 Overlaps and optimal eigenvalues
We restate the relation between the resolvent and the overlaps from the main text (18). For  $\tilde{\pmb{x}}_i = [x_i^\top, \mathbf{0}_M]^\top$  with  $\pmb{x}_i$  eigenvectors of  $\pmb{X}$ , we have:
$$
\tilde {\boldsymbol {x}} _ {i} ^ {\mathsf {T}} \left(\operatorname {I m} \boldsymbol {G} _ {S} (x - \mathrm {i} \epsilon)\right) \tilde {\boldsymbol {x}} _ {i} \approx \pi \bar {\mu} _ {S} (x) O _ {X} (x, \lambda_ {i}) \tag {47}
$$
From (47), (42), we find:
$$
\begin{array}{l} O _ {X} (\gamma , \lambda_ {i}) \approx \frac {1}{\pi \bar {\mu} _ {S} (\gamma)} \operatorname {I m} \lim  _ {z \rightarrow \gamma - \mathrm {i} 0 ^ {+}} \boldsymbol {x} _ {i} ^ {\intercal} \zeta_ {3} ^ {* - 1} \boldsymbol {G} _ {X ^ {2}} \left(\frac {z - \zeta_ {1} ^ {*}}{\zeta_ {3} ^ {*}}\right) \boldsymbol {x} _ {i} \tag {48} \\ = \frac {1}{\pi \bar {\mu} _ {S} (\gamma)} \operatorname {I m} \lim  _ {z \rightarrow \gamma - \mathrm {i} 0 ^ {+}} \frac {1}{z - \zeta_ {1} ^ {*} - \zeta_ {3} ^ {*} \lambda_ {i} ^ {2}} \\ \end{array}
$$
Once we have the overlap, we can compute the optimal eigenvalues from (14) in section 5. Note that, until now we had absorbed  $\sqrt{\kappa}$  into  $\mathbf{X}$ . Therefore, we should use (48) with  $O_{X}(\gamma, \sqrt{\kappa}\lambda_{i})$ . This leads to:
$$
\begin{array}{l} \widehat {\xi} _ {x i} ^ {*} \approx \frac {1}{N} \sum_ {j = 1} ^ {N} \lambda_ {j} O _ {X} (\gamma_ {i}, \sqrt {\kappa} \lambda_ {j}) \\ \approx \frac {1}{\pi \bar {\mu} _ {S} (\gamma_ {i})} \mathrm {I m} \lim _ {z \to \gamma_ {i} - \mathrm {i 0} ^ {+}} \frac {1}{N} \sum_ {j = 1} ^ {N} \frac {\lambda_ {j}}{z - \zeta_ {1} ^ {*} - \zeta_ {3} ^ {*} \kappa \lambda_ {j} ^ {2}} \\ = \frac {1}{\pi \bar {\mu} _ {S} (\gamma_ {i})} \operatorname {I m} \lim  _ {z \rightarrow \gamma_ {i} - \mathrm {i} 0 ^ {+}} \frac {1}{\kappa \zeta_ {3} ^ {*}} \frac {1}{N} \sum_ {j = 1} ^ {N} \frac {\lambda_ {j}}{\frac {z - \zeta_ {1} ^ {*}}{\kappa \zeta_ {3} ^ {*}} - \lambda_ {j} ^ {2}} \tag {49} \\ = \frac {1}{\kappa \pi \bar {\mu} _ {S} (\gamma_ {i})} \operatorname {I m} \lim  _ {z \rightarrow \gamma_ {i} - \mathrm {i} 0 ^ {+}} \frac {1}{\zeta_ {3} ^ {*}} \left(\frac {1}{2} \frac {1}{N} \sum_ {j = 1} ^ {N} \frac {1}{\sqrt {\frac {z - \zeta_ {1} ^ {*}}{\kappa \zeta_ {3} ^ {*}}} - \lambda_ {j}} - \frac {1}{2} \frac {1}{N} \sum_ {j = 1} ^ {N} \frac {1}{\sqrt {\frac {z - \zeta_ {1} ^ {*}}{\kappa \zeta_ {3} ^ {*}}} + \lambda_ {j}}\right) \\ \approx \frac {1}{\kappa \pi \bar {\mu} _ {S} (\gamma_ {i})} \mathrm {I m} \lim _ {z \to \gamma_ {i} - \mathrm {i 0} ^ {+}} \left\{\frac {1}{2} \frac {1}{\zeta_ {3} ^ {*}} \mathcal {G} _ {\rho_ {X}} \Big (\sqrt {\frac {z - \zeta_ {1} ^ {*}}{\kappa \zeta_ {3} ^ {*}}} \Big) - \frac {1}{2} \frac {1}{\zeta_ {3} ^ {*}} \mathcal {G} _ {\rho_ {- X}} \Big (\sqrt {\frac {z - \zeta_ {1} ^ {*}}{\kappa \zeta_ {3} ^ {*}}} \Big) \right\} \\ = \frac {1}{2 \kappa \pi \bar {\mu} _ {S} (\gamma_ {i})} \operatorname {I m} \lim  _ {z \rightarrow \gamma_ {i} - \mathrm {i} 0 ^ {+}} \left\{\frac {1}{\zeta_ {3} ^ {*}} \left[ \mathcal {G} _ {\rho_ {X}} \left(\sqrt {\frac {z - \zeta_ {1} ^ {*}}{\kappa \zeta_ {3} ^ {*}}}\right) + \mathcal {G} _ {\rho_ {X}} \left(- \sqrt {\frac {z - \zeta_ {1} ^ {*}}{\kappa \zeta_ {3} ^ {*}}}\right)\right]\right\} \\ \end{array}
$$
C.2.1 Estimating  $X^2$
The resolvent relation we have found in (42) is in terms of  $G_{X^2}$ . Therefore, like other RIEs in other problems [14, 19], we can express the estimator for  $X^2$  without any knowledge about  $\rho_X$  or  $\rho_{X^2}$ . One can see that, the optimal RIE for  $X^2$  is constructed in the same way as for  $X$  with eigenvalues denoted by  $\widehat{\xi_{x^2i}^*}$ . To compute the optimal eigenvalues, we absorb  $\sqrt{\kappa}$  into  $X$  and we use the exact expression in (48). In the end, we only need to divide by  $\kappa$  to find an estimator for the true  $X^2$ .
$$
\begin{array}{l} \widehat {\xi_ {x ^ {2}   i} ^ {*}} \approx \frac {1}{N} \sum_ {j = 1} ^ {N} \lambda_ {j} ^ {2} O _ {X} (\gamma_ {i}, \lambda_ {j}) \\ \approx \frac {1}{\pi \bar {\mu} _ {S} (\gamma_ {i})} \operatorname {I m} \lim  _ {z \to \gamma_ {i} - \mathrm {i} 0 ^ {+}} \frac {1}{N} \sum_ {j = 1} ^ {N} \frac {\lambda_ {j} ^ {2}}{z - \zeta_ {1} ^ {*} - \zeta_ {3} ^ {*} \lambda_ {j} ^ {2}} \\ = \frac {1}{\pi \bar {\mu} _ {S} (\gamma_ {i})} \operatorname {I m} \lim  _ {z \rightarrow \gamma_ {i} - \mathrm {i} 0 ^ {+}} \frac {1}{\zeta_ {3} ^ {*}} \frac {1}{N} \sum_ {j = 1} ^ {N} \frac {\lambda_ {j} ^ {2}}{\frac {z - \zeta_ {1} ^ {*}}{\zeta_ {3} ^ {*}} - \lambda_ {j} ^ {2}} \\ = \frac {1}{\pi \bar {\mu} _ {S} \left(\gamma_ {i}\right)} \operatorname {I m} \lim  _ {z \rightarrow \gamma_ {i} - \mathrm {i} 0 ^ {+}} - \frac {1}{\zeta_ {3} ^ {*}} \frac {1}{N} \sum_ {j = 1} ^ {N} \frac {\frac {z - \zeta_ {1} ^ {*}}{\zeta_ {3} ^ {*}} - \lambda_ {j} ^ {2} - \frac {z - \zeta_ {1} ^ {*}}{\zeta_ {3} ^ {*}}}{\frac {z - \zeta_ {1} ^ {*}}{\zeta_ {3} ^ {*}} - \lambda_ {j} ^ {2}} \tag {50} \\ = \frac {1}{\pi \bar {\mu} _ {S} (\gamma_ {i})} \operatorname {I m} \lim  _ {z \rightarrow \gamma_ {i} - \mathrm {i} 0 ^ {+}} - \frac {1}{\zeta_ {3} ^ {*}} \frac {1}{N} \sum_ {j = 1} ^ {N} \left[ 1 - \frac {z - \zeta_ {1} ^ {*}}{\zeta_ {3} ^ {*}} \frac {1}{\frac {z - \zeta_ {1} ^ {*}}{\zeta_ {3} ^ {*}} - \lambda_ {j} ^ {2}} \right] \\ \approx \frac {1}{\pi \bar {\mu} _ {S} (\gamma_ {i})} \operatorname {I m} \lim  _ {z \rightarrow \gamma_ {i} - \mathrm {i} 0 ^ {+}} - \frac {1}{\zeta_ {3} ^ {*}} + \frac {z - \zeta_ {1} ^ {*}}{\zeta_ {3} ^ {* 2}} \mathcal {G} _ {\rho_ {X 2}} \left(\frac {z - \zeta_ {1} ^ {*}}{\zeta_ {3} ^ {*}}\right) \\ \stackrel {\mathrm {(a)}} {=} \frac {1}{\pi \bar {\mu} _ {S} (\gamma_ {i})} \operatorname {I m} \lim  _ {z \rightarrow \gamma_ {i} - \mathrm {i} 0 ^ {+}} p _ {3} ^ {*} \\ \stackrel {(b)} {=} \frac {1}{\pi \bar {\mu} _ {S} (\gamma_ {i})} \operatorname {I m} \lim  _ {z \rightarrow \gamma_ {i} - \mathrm {i} 0 ^ {+}} \frac {z - \zeta_ {1} ^ {*}}{\zeta_ {3} ^ {*}} \mathcal {G} _ {\bar {\mu} _ {S}} (z) - \frac {1}{\zeta_ {3} ^ {*}} \\ \end{array}
$$
where in (a) we used (41), and for (b) we used (45). Thus, the optimal eigenvalues for  $X^2$  read:
$$
\widehat {\xi_ {x ^ {2} i} ^ {*}} = \frac {1}{\kappa} \frac {1}{\pi \bar {\mu} _ {S} (\gamma_ {i})} \operatorname {I m} \lim  _ {z \rightarrow \gamma_ {i} - \mathrm {i} 0 ^ {+}} \frac {z - \zeta_ {1} ^ {*}}{\zeta_ {3} ^ {*}} \mathcal {G} _ {\bar {\mu} _ {S}} (z) - \frac {1}{\zeta_ {3} ^ {*}} \tag {51}
$$
Note that the parameters  $\zeta_1^*$ ,  $\zeta_3^*$  can be computed from (45), (46), without the knowledge of  $\rho_X$  or  $\rho_{X^2}$ .
Remark 3. The main barrier to find an estimator for  $\mathbf{X}$  is that the resolvent relation (42) is in terms of  $\mathcal{G}_{\rho_{X^2}}$ . Moreover, in the estimator for  $\mathbf{X}$ , second equality in (49), we have the sum  $\sum_{j=1}^{N} \frac{\lambda_j}{z - \zeta_1^* - \kappa \zeta_3^* \lambda_j^2}$  which cannot be written in terms of  $\mathcal{G}_{\rho_{X^2}}$ .
Remark 4. If we add the assumption that the matrix  $\mathbf{X}$  is positive semi-definite, without any further knowledge on the prior, we can use  $\sqrt{\widehat{\xi_{x^2 i}^*}}$  for the eigenvalues of  $\Xi_X(S)$ . However, note that, this estimator is sub-optimal for  $\mathbf{X}$  as  $\sqrt{\sum_{j=1}^{N} \lambda_j^2 (\mathbf{u}_i^\top \mathbf{x}_j)^2} \neq \sum_{j=1}^{N} \lambda_j (\mathbf{u}_i^\top \mathbf{x}_j)^2$ .
C.3 Numerical Examples
In this section, we will illustrate the derived formulas (42), (48), and (49) with numerical experiments.
We consider matrices  $\mathbf{Y}$ ,  $\mathbf{W} \in \mathbb{R}^{N \times M}$  to have i.i.d. Gaussian entries, so  $\mathcal{C}_{\mu_Y}^{(\alpha)}(z) = \mathcal{C}_{\mu_W}^{(\alpha)}(z) = \frac{1}{\alpha} z$  which leads to a simplification of saddle point equations (45):
$$
\left\{ \begin{array}{l} \zeta_ {1} ^ {*} = \frac {1}{\alpha} p _ {2} ^ {*}, \quad \zeta_ {2} ^ {*} = \alpha z \frac {z \mathcal {G} _ {\bar {\mu} _ {S}} (z) - 1}{\alpha z \mathcal {G} _ {\bar {\mu} _ {S}} (z) + 1 - \alpha}, \quad \zeta_ {3} ^ {*} = \frac {1}{\alpha} p _ {2} ^ {*} \\ p _ {1} ^ {*} = \mathcal {G} _ {\bar {\mu} _ {S}} (z), \quad p _ {2} ^ {*} = \alpha \mathcal {G} _ {\bar {\mu} _ {S}} (z) + (1 - \alpha) \frac {1}{z}, \quad p _ {3} ^ {*} = \frac {z - \zeta_ {1} ^ {*}}{\zeta_ {3} ^ {*}} \mathcal {G} _ {\bar {\mu} _ {S}} (z) - \frac {1}{\zeta_ {3} ^ {*}} \end{array} \right. \tag {52}
$$
C.3.1 Resolvent relation
We take  $\kappa = 1$ . In model (29), without loss of generality we can consider  $\mathbf{X}$  to be diagonal. In figures 5 and 6 respectively, we consider the  $\mathbf{X}$  to be a diagonal matrix obtained by taking the eigenvalues of a Wigner matrix and a Wishart matrix respectively.
Note that  $\mu_{S}$  and  $\mathcal{G}_{\bar{\mu}_S}(z)$  can be computed analytically using tools from random matrix theory, but the computation is highly involved. In our experiments, we use instead a numerical estimation of  $\mathcal{G}_{\bar{\mu}_S}(z)$  obtained from the observation matrix with the help of a Cauchy kernel to compute the parameters  $\zeta_1^*,\zeta_3^*$  (see section G, and [58] for details on the Cauchy kernel method).
Unlike the simpler models [15] for which the fluctuations are derived to be of the order  $\frac{1}{\sqrt{N}}$ , based on our derivation we cannot assess the order of fluctuations. However, from our numerics we observe that the fluctuations are of the order  $o(N)$ . Moreover, fluctuations near the edge points of density are larger (in particular for the last row in both figures 5, 6), which is due to the fact that the limiting measures have higher fluctuations on their edge-points.
Another observation, from comparison of figures 5, 6, is that the fluctuations for the first example are relatively larger than the second one. One possible guess could be that this is due to the symmetry of  $\rho_{X}$  in the first example. However based on more extensive numerical observations (which we omit here) we speculate that this issue is in fact related to the existence of small eigenvalues of  $X$ . In other words, if  $X$  has eigenvalue 0 or small eigenvalues, we have higher fluctuations in the relation (47).
C.3.2 Overlaps
To illustrate the formula for the overlap (48), we fix the matrix  $\mathbf{X}$  and run experiments over various realization of the model (29). For each experiment, we record the overlap of  $k$ -th left singular vector of  $S$  and the eigenvectors of  $\mathbf{X}$ . To compute the theoretical prediction, we find  $\zeta_1^* = \zeta_3^*$  for  $z = \bar{\gamma}_k - \mathrm{i}0^+$  where  $\bar{\gamma}_k$  is the average of  $k$ -th singular value of  $S$  in the experiments.
To find  $\zeta_1^* = \zeta_3^*$ , we use the set of equations (41) which for  $Y$ ,  $W$  Gaussian can be written as:
$$
\left\{ \begin{array}{l} \zeta_ {1} ^ {*} = \frac {1}{\alpha} p _ {2} ^ {*}, \quad \zeta_ {2} ^ {*} = p _ {1} ^ {*} + p _ {3} ^ {*}, \quad \zeta_ {3} ^ {*} = \frac {1}{\alpha} p _ {2} ^ {*} \\ p _ {1} ^ {*} = \frac {1}{\zeta_ {1} ^ {*}} \mathcal {G} _ {\rho_ {X ^ {2}}} \left(\frac {z}{\zeta_ {1} ^ {*}} - 1\right), \quad p _ {2} ^ {*} = \frac {1}{z - \zeta_ {2} ^ {*}}, \quad p _ {3} ^ {*} = \frac {z - \zeta_ {1} ^ {*}}{\zeta_ {1} ^ {* 2}} \mathcal {G} _ {\rho_ {X ^ {2}}} \left(\frac {z}{\zeta_ {1} ^ {*}} - 1\right) - \frac {1}{\zeta_ {1} ^ {*}} \end{array} \right. \tag {53}
$$
Now we proceed to simplify the solution above:
$$
\zeta_ {2} ^ {*} = p _ {1} ^ {*} + p _ {3} ^ {*} = \frac {z}{\zeta_ {1} ^ {* 2}} \mathcal {G} _ {\rho_ {X ^ {2}}} \left(\frac {z}{\zeta_ {1} ^ {*}} - 1\right) - \frac {1}{\zeta_ {1} ^ {*}}
$$
(a) Entry  $i = j = 1$ , first diagonal entry in the upper-left block
(b) Entry  $i = j = 5000$ , last diagonal entry in the lower-right block
(c) Entry  $i = 4999$ ,  $j = 5000$ , a non-diagonal entry in the lower-right block
(d) Entry  $i = 1, j = 5000$ , an entry in the upper-right block
(e) Entry  $i = 1, j = 2$ , a non-diagonal entry in the upper-left block
Figure 5: Illustration of (42).  $\mathbf{X}$  is diagonal matrix from the eigenvalues of a Wigner matrix and  $\mathbf{Y}, \mathbf{Z}$  are Gaussian matrices with  $N = 2000$ ,  $M = 3000$ . The empirical estimate of  $G_{\mathcal{S}}(z)$  (dashed blue line) is computed for  $z = \gamma_i - \mathrm{i}\sqrt{\frac{1}{2N}}$  for  $1 \leq i \leq N$ . Theoretical estimate (solid orange line) computed from the rhs of (42) with parameters obtained from the generated matrix. Note that, the theoretical estimate has also fluctuations because the parameters  $\zeta_1^*, \zeta_3^*$  are given by the numerical estimate of  $\mathcal{G}_{\bar{\mu}_S}(z)$ .