SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \langle G _ {i j} (z) \rangle = \lim  _ {n \rightarrow \infty} \int \left(\prod_ {k = 1} ^ {N + M} \prod_ {\tau = 1} ^ {n} d \eta_ {k} ^ {(\tau)}\right) \eta_ {i} ^ {(1)} \eta_ {j} ^ {(1)} \\ \times \exp \left\{- \frac {1}{2} \sum_ {\tau = 1} ^ {n} \left[ z \| \boldsymbol {\eta} ^ {(\tau)} \| ^ {2} - N \mathcal {Q} _ {\mu W} ^ {(\alpha)} \left(\frac {1}{N M} \| \boldsymbol {a} ^ {(\tau)} \| ^ {2} \| \boldsymbol {b} ^ {(\tau)} \| ^ {2}\right) \right. \right. \tag {65} \\ - N \mathcal {P} _ {\rho_ {X}} \left(\frac {1}{N} \left(\boldsymbol {a} ^ {(\tau) \top} \boldsymbol {Y} \boldsymbol {b} ^ {(\tau)} + \| \boldsymbol {a} ^ {(\tau)} \| \| \boldsymbol {Y} \boldsymbol {b} ^ {(\tau)} \|\right)\right) \\ \left. \left. - N \mathcal {P} _ {\rho_ {X}} \left(\frac {1}{N} \left(\boldsymbol {a} ^ {(\tau) \intercal} \boldsymbol {Y} \boldsymbol {b} ^ {(\tau)} - \| \boldsymbol {a} ^ {(\tau)} \| \| \boldsymbol {Y} \boldsymbol {b} ^ {(\tau)} \|\right)\right) \right] \right\} \\ \end{array}
$$
Now, we introduce delta functions (for brevity we drop the limit term):
$$
\langle G _ {i j} (z) \rangle = \int \left(\prod_ {k = 1} ^ {N + M} \prod_ {\tau = 1} ^ {n} d \eta_ {k} ^ {(\tau)}\right) \left(\prod_ {\tau = 1} ^ {n} d p _ {1} ^ {(\tau)} d q _ {2} ^ {(\tau)} d q _ {3} ^ {(\tau)} d q _ {4} ^ {(\tau)}\right) \eta_ {i} ^ {(1)} \eta_ {j} ^ {(1)}
$$
$$
\begin{array}{l} \times \prod_ {\tau = 1} ^ {n} \delta \left(q _ {1} ^ {(\tau)} - \frac {1}{N} \| \boldsymbol {a} ^ {(\tau)} \| ^ {2}\right) \delta \left(q _ {2} ^ {(\tau)} - \frac {1}{M} \| \boldsymbol {b} ^ {(\tau)} \| ^ {2}\right) \\ \times \delta \left(q _ {3} ^ {(\tau)} - \frac {1}{N} \| \boldsymbol {Y} \boldsymbol {b} ^ {(\tau)} \| ^ {2}\right) \delta \left(q _ {4} ^ {(\tau)} - \frac {1}{N} \boldsymbol {a} ^ {(\tau) \top} \boldsymbol {Y} \boldsymbol {b} ^ {(\tau)}\right) \tag {66} \\ \end{array}
$$
$$
\begin{array}{l} \times \exp \left\{- \frac {1}{2} \sum_ {\tau = 1} ^ {n} z \| \boldsymbol {\eta} ^ {(\tau)} \| ^ {2} - N \mathcal {Q} _ {\mu_ {W}} ^ {(\alpha)} \left(q _ {1} ^ {(\tau)} q _ {2} ^ {(\tau)}\right) \right. \\ - N \mathcal {P} _ {\rho_ {X}} \left(q _ {4} ^ {(\tau)} + \sqrt {q _ {1} ^ {(\tau)} q _ {3} ^ {(\tau)}}\right) - N \mathcal {P} _ {\rho_ {X}} \left(q _ {4} ^ {(\tau)} - \sqrt {q _ {1} ^ {(\tau)} q _ {3} ^ {(\tau)}}\right) \Bigg \} \\ \end{array}
$$
In the next step, we replace each delta with its Fourier transform. Note that for the parameters  $q_{1}, q_{2}, q_{3}$  we use  $\delta\left(q_{1}^{\tau}-\frac{1}{N}\|\pmb{a}^{\tau}\|^{2}\right) \propto \int d\beta_{1}^{\tau}\exp\left\{-\frac{N}{2}\beta_{1}^{\tau}\left(q_{1}^{\tau}-\frac{1}{N}\|\pmb{a}^{\tau}\|^{2}\right)\right\}$ , and for  $q_{4}$  we use  $\delta\left(q_{4}^{(\tau)}-\frac{1}{N}\pmb{a}^{(\tau)^{\mathsf{T}}}\pmb{Y}\pmb{b}^{(\tau)}\right) \propto \int d\beta_{1}^{\tau}\exp\left\{-N\beta_{1}^{\tau}\left(q_{4}^{(\tau)}-\frac{1}{N}\pmb{a}^{(\tau)^{\mathsf{T}}}\pmb{Y}\pmb{b}^{(\tau)}\right)\right\}$ . After rearranging, we find:
$$
\begin{array}{l} \langle G _ {i j} (z) \rangle \propto \int \big (\prod_ {\tau = 1} ^ {n} d q _ {1} ^ {(\tau)} d q _ {2} ^ {(\tau)} d q _ {3} ^ {(\tau)} d q _ {4} ^ {(\tau)} d \beta_ {1} ^ {(\tau)} d \beta_ {2} ^ {(\tau)} d \beta_ {3} ^ {(\tau)} d \beta_ {4} ^ {(\tau)} \big) \\ \times \exp \left\{\frac {N}{2} \sum_ {\tau = 1} ^ {n} \mathcal {Q} _ {\mu_ {W}} ^ {(\alpha)} (q _ {1} ^ {(\tau)} q _ {2} ^ {(\tau)}) + \mathcal {P} _ {\rho_ {X}} \left(q _ {4} ^ {(\tau)} + \sqrt {q _ {1} ^ {(\tau)} q _ {3} ^ {(\tau)}}\right) + \mathcal {P} _ {\rho_ {X}} \left(q _ {4} ^ {(\tau)} - \sqrt {q _ {1} ^ {(\tau)} q _ {3} ^ {(\tau)}}\right) \right. \\ \left. - \beta_ {1} ^ {(\tau)} q _ {1} ^ {(\tau)} - \frac {1}{\alpha} \beta_ {2} ^ {(\tau)} q _ {2} ^ {(\tau)} - \beta_ {3} ^ {(\tau)} q _ {3} ^ {(\tau)} - 2 \beta_ {4} ^ {(\tau)} q _ {4} ^ {(\tau)} \right\} \\ \times \int \left(\prod_ {k = 1} ^ {N + M} \prod_ {\tau = 1} ^ {n} d \eta_ {k} ^ {(\tau)}\right) \eta_ {i} ^ {(1)} \eta_ {j} ^ {(1)} \exp \left\{- \frac {1}{2} \sum_ {\tau = 1} ^ {n} z \| \boldsymbol {\eta} ^ {(\tau)} \| - \beta_ {1} ^ {(\tau)} \| \boldsymbol {a} ^ {(\tau)} \| ^ {2} - \beta_ {2} ^ {(\tau)} \| \boldsymbol {b} ^ {(\tau)} \| ^ {2} \right. \\ \left. - \beta_ {3} ^ {(\tau)} \| \boldsymbol {Y} \boldsymbol {b} ^ {(\tau)} \| ^ {2} - 2 \beta_ {4} ^ {(\tau)} \boldsymbol {a} ^ {(\tau) \mathsf {T}} \boldsymbol {Y} \boldsymbol {b} ^ {(\tau)} \right\} \tag {67} \\ \end{array}
$$
The inner integral is a Gaussian integral, and can be written as:
$$
\begin{array}{l} \int \left(\prod_ {k = 1} ^ {N + M} \prod_ {\tau = 1} ^ {n} d \eta_ {k} ^ {(\tau)}\right) \eta_ {i} ^ {(1)} \eta_ {j} ^ {(1)} \\ \times \exp \left\{\sum_ {\tau = 1} ^ {n} - \frac {1}{2} \boldsymbol {\eta} ^ {(\tau) \intercal} \left[ \begin{array}{c c} (z - \beta_ {1} ^ {(\tau)}) \boldsymbol {I} _ {N} & - \beta_ {4} ^ {(\tau)} \boldsymbol {Y} \\ - \beta_ {4} ^ {(\tau)} \boldsymbol {Y} ^ {\intercal} & (z - \beta_ {2} ^ {(\tau)}) \boldsymbol {I} _ {M} - \beta_ {3} ^ {(\tau)} \boldsymbol {Y} ^ {\intercal} \boldsymbol {Y} \end{array} \right] \boldsymbol {\eta} ^ {(\tau)} \right\} \tag {68} \\ \end{array}
$$
Denote the matrix in the exponent by  $C_Y^{(\tau)}$ . Using the formula for determinant of block matrices (see proposition 2.8.4 in [61]), we have:
$$
\begin{array}{l} \det \boldsymbol {C} _ {Y} ^ {(\tau)} = \det \left[ (z - \beta_ {1} ^ {(\tau)}) \boldsymbol {I} _ {N} - \beta_ {4} ^ {(\tau) ^ {2}} \boldsymbol {Y} \big ((z - \beta_ {2} ^ {(\tau)}) \boldsymbol {I} _ {M} - \beta_ {3} ^ {(\tau)} \boldsymbol {Y} ^ {\intercal} \boldsymbol {Y} \big) ^ {- 1} \boldsymbol {Y} ^ {\intercal} \right] \\ \times \det  \left[ (z - \beta_ {2} ^ {(\tau)}) \boldsymbol {I} _ {M} - \beta_ {3} ^ {(\tau)} \boldsymbol {Y} ^ {\intercal} \boldsymbol {Y} \right] \\ = \prod_ {k = 1} ^ {N} \left[ z - \beta_ {1} ^ {(\tau)} - \beta_ {4} ^ {(\tau) ^ {2}} \frac {\sigma_ {k} ^ {2}}{z - \beta_ {2} ^ {(\tau)} - \beta_ {3} ^ {(\tau)} \sigma_ {k} ^ {2}} \right] \prod_ {k = 1} ^ {N} \left(z - \beta_ {2} ^ {(\tau)} - \beta_ {3} ^ {(\tau)} \sigma_ {k} ^ {2}\right) \left(z - \beta_ {2} ^ {(\tau)}\right) ^ {M - N} \\ = \left(z - \beta_ {2} ^ {(\tau)}\right) ^ {M - N} \prod_ {k = 1} ^ {N} \left[ \left(z - \beta_ {1} ^ {(\tau)}\right) \left(z - \beta_ {2} ^ {(\tau)} - \beta_ {3} ^ {(\tau)} \sigma_ {k} ^ {2}\right) - \beta_ {4} ^ {(\tau) ^ {2}} \sigma_ {k} ^ {2} \right] \\ = \left(z - \beta_ {2} ^ {(\tau)}\right) ^ {M - N} \prod_ {k = 1} ^ {N} \left[ \left(z - \beta_ {1} ^ {(\tau)}\right) \left(z - \beta_ {2} ^ {(\tau)}\right) - \left(\beta_ {4} ^ {(\tau) ^ {2}} + \beta_ {3} ^ {(\tau)} \left(z - \beta_ {1} ^ {(\tau)}\right)\right) \sigma_ {k} ^ {2} \right] \\ \end{array}
$$
where  $\sigma_{k}$ 's are the singular values of  $\pmb{Y}$ . So computing the Gaussian integrals, (67) can be written as:
$$
\begin{array}{l} \langle G _ {i j} (z) \rangle \propto \int \left(\prod_ {\tau = 1} ^ {n} d q _ {1} ^ {(\tau)} d q _ {2} ^ {(\tau)} d q _ {3} ^ {(\tau)} d q _ {4} ^ {(\tau)} d \beta_ {1} ^ {(\tau)} d \beta_ {2} ^ {(\tau)} d \beta_ {3} ^ {(\tau)} d \beta_ {4} ^ {(\tau)}\right) \left(C _ {Y} ^ {(1) ^ {- 1}}\right) _ {i j} \\ \times \exp \left\{- \frac {N n}{2} F _ {0} ^ {Y} \left(\boldsymbol {q} _ {1}, \boldsymbol {q} _ {2}, \boldsymbol {q} _ {3}, \boldsymbol {q} _ {4}, \beta_ {1}, \beta_ {2}, \beta_ {3}, \beta_ {4}\right) \right\} \tag {69} \\ \end{array}
$$
$$
\begin{array}{l} F _ {0} ^ {Y} \left(\boldsymbol {q} _ {1}, \boldsymbol {q} _ {2}, \boldsymbol {q} _ {3}, \boldsymbol {q} _ {4}, \boldsymbol {\beta} _ {1}, \boldsymbol {\beta} _ {2}, \boldsymbol {\beta} _ {3}, \boldsymbol {\beta} _ {4}\right) = \frac {1}{n} \sum_ {\tau = 1} ^ {n} \left[ \left(\frac {1}{\alpha} - 1\right) \ln \left(z - \beta_ {2} ^ {(\tau)}\right) \right. \\ + \frac {1}{N} \sum_ {k = 1} ^ {N} \ln \left((z - \beta_ {1} ^ {(\tau)}) (z - \beta_ {2} ^ {(\tau)}) - \left(\beta_ {4} ^ {(\tau) ^ {2}} + \beta_ {3} ^ {(\tau)} (z - \beta_ {1} ^ {(\tau)})\right) \sigma_ {k} ^ {2}\right) \tag {70} \\ - \mathcal {Q} _ {\mu w} ^ {(\alpha)} (q _ {1} ^ {(\tau)} q _ {2} ^ {(\tau)}) - \mathcal {P} _ {\rho_ {X}} (q _ {4} ^ {(\tau)} + \sqrt {q _ {1} ^ {(\tau)} q _ {3} ^ {(\tau)}}) - \mathcal {P} _ {\rho_ {X}} (q _ {4} ^ {(\tau)} - \sqrt {q _ {1} ^ {(\tau)} q _ {3} ^ {(\tau)}}) \\ \left. + \beta_ {1} ^ {(\tau)} q _ {1} ^ {(\tau)} + \frac {1}{\alpha} \beta_ {2} ^ {(\tau)} q _ {2} ^ {(\tau)} + \beta_ {3} ^ {(\tau)} q _ {3} ^ {(\tau)} + 2 \beta_ {4} ^ {(\tau)} q _ {4} ^ {(\tau)} \right] \\ \end{array}
$$
We will evaluate the integral (67) using saddle-points of the function  $F_0^Y$ . From the replica symmetric ansatz at the saddle-point we have:
$$
\forall \tau \in \{1, \dots , n \}: \quad \left\{ \begin{array}{l l} q _ {1} ^ {\tau} = q _ {1}, & q _ {2} ^ {\tau} = q _ {2}, \quad q _ {3} ^ {\tau} = q _ {3}, \quad q _ {4} ^ {\tau} = q _ {4} \\ \beta_ {1} ^ {\tau} = \beta_ {1}, & \beta_ {2} ^ {\tau} = \beta_ {2}, \quad \beta_ {3} ^ {\tau} = \beta_ {3}, \quad \beta_ {4} ^ {\tau} = \beta_ {4} \end{array} \right.
$$
Finally, we find the solution to be:
$$
\left\{\begin{array}{l}\beta_ {1} ^ {*} = \frac {\mathcal {C} _ {\mu W} ^ {(\alpha)} \left(q _ {1} ^ {*} q _ {2} ^ {*}\right)}{q _ {1} ^ {*}} + \frac {1}{2} \sqrt {\frac {q _ {3} ^ {*}}{q _ {1} ^ {*}}} \left(\mathcal {R} _ {\rho_ {X}} \left(q _ {4} ^ {*} + \sqrt {q _ {1} ^ {*} q _ {3} ^ {*}}\right) - \mathcal {R} _ {\rho_ {X}} \left(q _ {4} ^ {*} - \sqrt {q _ {1} ^ {*} q _ {3} ^ {*}}\right)\right)\\\beta_ {2} ^ {*} = \alpha \frac {\mathcal {C} _ {\mu W} ^ {(\alpha)} \left(q _ {1} ^ {*} q _ {2} ^ {*}\right)}{q _ {2} ^ {*}}\\\beta_ {3} ^ {*} = \frac {1}{2} \sqrt {\frac {q _ {1} ^ {*}}{q _ {3} ^ {*}}} \left(\mathcal {R} _ {\rho_ {X}} \left(q _ {4} ^ {*} + \sqrt {q _ {1} ^ {*} q _ {3} ^ {*}}\right) - \mathcal {R} _ {\rho_ {X}} \left(q _ {4} ^ {*} - \sqrt {q _ {1} ^ {*} q _ {3} ^ {*}}\right)\right)\\\left. \right. \beta_ {4} ^ {*} = \frac {1}{2} \left(\mathcal {R} _ {\rho_ {X}} \left(q _ {4} ^ {*} + \sqrt {q _ {1} ^ {*} q _ {3} ^ {*}}\right) + \mathcal {R} _ {\rho_ {X}} \left(q _ {4} ^ {*} - \sqrt {q _ {1} ^ {*} q _ {3} ^ {*}}\right)\right) \quad \text {w i t h} \left\{\begin{array}{l}Z _ {1} (z) = (z - \beta_ {1} ^ {*}) (z - \beta_ {2} ^ {*})\\Z _ {2} (z) = \beta_ {4} ^ {* 2} + \beta_ {3} ^ {*} (z - \beta_ {1} ^ {*})\end{array}\right.\\q _ {1} ^ {*} = \frac {(z - \beta_ {2} ^ {*}) \beta_ {4} ^ {* 2}}{Z _ {2} (z) ^ {2}} \mathcal {G} _ {\rho_ {Y}} \left(\frac {Z _ {1} (z)}{Z _ {2} (z)}\right) + \frac {\beta_ {3} ^ {*}}{Z _ {2} (z)}\\q _ {2} ^ {*} = \alpha \frac {z - \beta_ {1} ^ {*}}{Z _ {2} (z)} \mathcal {G} _ {\rho_ {Y}} \left(\frac {Z _ {1} (z)}{Z _ {2} (z)}\right) + \frac {1 - \alpha}{z - \beta_ {2} ^ {*}}\\q _ {3} ^ {*} = \frac {(z - \beta_ {1} ^ {*}) Z _ {1} (z)}{Z _ {2} (z) ^ {2}} \mathcal {G} _ {\rho_ {Y}} \left(\frac {Z _ {1} (z)}{Z _ {2} (z)}\right) - \frac {z - \beta_ {1} ^ {*}}{Z _ {2} (z)}\\q _ {4} ^ {*} = \frac {\beta_ {4} ^ {*} Z _ {1} (z)}{Z _ {2} (z) ^ {2}} \mathcal {G} _ {\rho_ {Y}} \left(\frac {Z _ {1} (z)}{Z _ {2} (z)}\right) - \frac {\beta_ {4} ^ {*}}{Z _ {2} (z)}\end{array}\right. \tag {71}
$$
where  $\rho_{Y}$  is the limiting eigenvalue distribution of  $\mathbf{Y}\mathbf{Y}^{\mathrm{T}}$ .
The relation (69) and the solutions (71) hold for arbitrary indices  $i, j$ , so we can state the relation in the matrix form. Computing the inverse of  $C_Y^{* - 1}$  (see section H.2), we have:
$$
\begin{array}{l} \left\langle \boldsymbol {G} _ {\mathcal {S}} (z) \right\rangle_ {\boldsymbol {O}, \boldsymbol {U}, \boldsymbol {V}} = \left\langle \left[ \begin{array}{c c} \frac {1}{z} \boldsymbol {I} _ {N} + \frac {1}{z} \boldsymbol {S} \boldsymbol {G} _ {S ^ {\intercal} S} (z ^ {2}) \boldsymbol {S} ^ {\intercal} & \boldsymbol {S} \boldsymbol {G} _ {S ^ {\intercal} S} (z ^ {2}) \\ \boldsymbol {G} _ {S ^ {\intercal} S} (z ^ {2}) \boldsymbol {S} ^ {\intercal} & z \boldsymbol {G} _ {S ^ {\intercal} S} (z ^ {2}) \end{array} \right] \right\rangle \\ = \left[ \begin{array}{c c} \frac {1}{z - \beta_ {1} ^ {*}} \boldsymbol {I} _ {N} + \frac {\beta_ {4} ^ {* 2}}{\left(z - \beta_ {1} ^ {*}\right) Z _ {2} (z)} \boldsymbol {Y} \boldsymbol {G} _ {Y \tau Y} \left(\frac {Z _ {1} (z)}{Z _ {2} (z)}\right) \boldsymbol {Y} ^ {\intercal} & \frac {\beta_ {4} ^ {*}}{Z _ {2} (z)} \boldsymbol {Y} \boldsymbol {G} _ {Y \tau Y} \left(\frac {Z _ {1} (z)}{Z _ {2} (z)}\right) \\ \frac {\beta_ {4} ^ {*}}{Z _ {2} (z)} \boldsymbol {G} _ {Y \tau Y} \left(\frac {Z _ {1} (z)}{Z _ {2} (z)}\right) \boldsymbol {Y} ^ {\intercal} & \frac {z - \beta_ {1} ^ {*}}{Z _ {2} (z)} \boldsymbol {G} _ {Y \tau Y} \left(\frac {Z _ {1} (z)}{Z _ {2} (z)}\right) \end{array} \right] \tag {72} \\ \end{array}
$$
With this relation, we can further simplify the solution (71).
We start with comparing the trace of upper-left block in (72). The normalized trace of the first block in  $\langle G_S(z) \rangle_{O,U,V}$  is computed in (43) to be  $\mathcal{G}_{\bar{\mu}_S}(z)$ . The normalized trace of the upper-left block in
$C_Y^*$  -1 is:
$$
\begin{array}{l} \frac {1}{N} \operatorname {T r} \left[ (z - \beta_ {1} ^ {*}) ^ {- 1} \boldsymbol {I} _ {N} + \frac {\beta_ {4} ^ {* 2}}{(z - \beta_ {1} ^ {*}) Z _ {2} (z)} \boldsymbol {Y} \boldsymbol {G} _ {Y ^ {\intercal} Y} \left(\frac {Z _ {1} (z)}{Z _ {2} (z)}\right) \boldsymbol {Y} ^ {\intercal} \right] \\ = \frac {1}{N} \frac {1}{z - \beta_ {1} ^ {*}} \sum_ {k = 1} ^ {N} \left[ 1 + \frac {\beta_ {4} ^ {* 2}}{Z _ {2} (z)} \frac {\sigma_ {k} ^ {2}}{\frac {Z _ {1} (z)}{Z _ {2} (z)} - \sigma_ {k} ^ {2}} \right] \\ = \frac {1}{N} \frac {1}{z - \beta_ {1} ^ {*}} \sum_ {k = 1} ^ {N} \left[ \frac {\beta_ {4} ^ {* 2} Z _ {1} (z)}{Z _ {2} ^ {2} (z)} \frac {1}{\frac {Z _ {1} (z)}{Z _ {2} (z)} - \sigma_ {k} ^ {2}} + 1 - \frac {\beta_ {4} ^ {* 2}}{Z _ {2} (z)} \right] \tag {73} \\ = \frac {1}{N} \frac {1}{z - \beta_ {1} ^ {*}} \frac {\beta_ {4} ^ {* 2} Z _ {1} (z)}{Z _ {2} ^ {2} (z)} \sum_ {k = 1} ^ {N} \frac {1}{\frac {Z _ {1} (z)}{Z _ {2} (z)} - \sigma_ {k} ^ {2}} + \frac {1}{z - \beta_ {1} ^ {*}} \frac {\beta_ {3} ^ {*} (z - \beta_ {1} ^ {*})}{Z _ {2} (z)} \\ = \frac {(z - \beta_ {2} ^ {*}) \beta_ {4} ^ {* 2}}{Z _ {2} (z) ^ {2}} \mathcal {G} _ {\rho_ {Y}} \left(\frac {Z _ {1} (z)}{Z _ {2} (z)}\right) + \frac {\beta_ {3} ^ {*}}{Z _ {2} (z)} \\ = q _ {1} ^ {*} \\ \end{array}
$$
Thus,  $q_{1}^{*} = \mathcal{G}_{\bar{\mu}_{S}}(z)$
The normalized trace of the lower-right block of  $\left\langle G_{S}(z)\right\rangle_{\mathcal{O},\mathcal{U},\mathcal{V}}$  is  $\alpha \mathcal{G}_{\bar{\mu}_S}(z) + (1 - \alpha)\frac{1}{z}$  (see (44)). The normalized trace of the lower-right block in  $C_Y^{*^{-1}}$  is:
$$
\begin{array}{l} \frac {1}{M} \operatorname {T r} \left[ \frac {z - \beta_ {1} ^ {*}}{Z _ {2} (z)} G _ {Y \tau Y} (\frac {Z _ {1} (z)}{Z _ {2} (z)}) \right] = \frac {1}{M} \frac {z - \beta_ {1} ^ {*}}{Z _ {2} (z)} \sum_ {k = 1} ^ {N} \frac {1}{\frac {Z _ {1} (z)}{Z _ {2} (z)} - \sigma_ {k} ^ {2}} + \frac {M - N}{M} \frac {z - \beta_ {1} ^ {*}}{Z _ {2} (z)} \frac {Z _ {2} (z)}{Z _ {1} (z)} \\ = \frac {N}{M} \frac {1}{N} \frac {z - \beta_ {1} ^ {*}}{Z _ {2} (z)} \sum_ {k = 1} ^ {N} \frac {1}{\frac {Z _ {1} (z)}{Z _ {2} (z)} - \sigma_ {k} ^ {2}} + \frac {M - N}{M} \frac {z - \beta_ {1} ^ {*}}{Z _ {1} (z)} \tag {74} \\ = \alpha \frac {z - \beta_ {1} ^ {*}}{Z _ {2} (z)} \mathcal {G} _ {\rho_ {Y}} \big (\frac {Z _ {1} (z)}{Z _ {2} (z)} \big) + \frac {1 - \alpha}{z - \beta_ {2} ^ {*}} \\ = q _ {2} ^ {*} \\ \end{array}
$$
So,  $q_{2}^{*} = \alpha \mathcal{G}_{\bar{\mu}_{S}}(z) + (1 - \alpha)\frac{1}{z}$ .
With a bit of algebra, we can express the parameters  $q_3^*$ ,  $q_4^*$  in terms of  $q_1^*$ ,  $\beta_1^*$ ,  $\beta_4^*$ :
$$
q _ {3} ^ {*} = \frac {\left(z - \beta_ {1} ^ {*}\right) ^ {2}}{\beta_ {4} ^ {* 2}} q _ {1} ^ {*} - \frac {z - \beta_ {1} ^ {*}}{\beta_ {4} ^ {* 2}}, \quad q _ {4} ^ {*} = \frac {z - \beta_ {1} ^ {*}}{\beta_ {4} ^ {*}} q _ {1} ^ {*} - \frac {1}{\beta_ {4} ^ {*}} \tag {75}
$$
Therefore, the solution can be written without involving  $\mathcal{G}_{\rho_Y}$ , as:
$$
\left\{ \begin{array}{l} \beta_ {1} ^ {*} = \frac {\mathcal {C} _ {\mu W} ^ {(\alpha)} \left(q _ {1} ^ {*} q _ {2} ^ {*}\right)}{q _ {1} ^ {*}} + \frac {1}{2} \sqrt {\frac {q _ {3} ^ {*}}{q _ {1} ^ {*}}} \left(\mathcal {R} _ {\rho_ {X}} \left(q _ {4} ^ {*} + \sqrt {q _ {1} ^ {*} q _ {3} ^ {*}}\right) - \mathcal {R} _ {\rho_ {X}} \left(q _ {4} ^ {*} - \sqrt {q _ {1} ^ {*} q _ {3} ^ {*}}\right)\right) \\ \beta_ {2} ^ {*} = \alpha \frac {\mathcal {C} _ {\mu W} ^ {(\alpha)} \left(q _ {1} ^ {*} q _ {2} ^ {*}\right)}{q _ {2} ^ {*}} \\ \beta_ {3} ^ {*} = \frac {1}{2} \sqrt {\frac {q _ {1} ^ {*}}{q _ {3} ^ {*}}} \left(\mathcal {R} _ {\rho_ {X}} \left(q _ {4} ^ {*} + \sqrt {q _ {1} ^ {*} q _ {3} ^ {*}}\right) - \mathcal {R} _ {\rho_ {X}} \left(q _ {4} ^ {*} - \sqrt {q _ {1} ^ {*} q _ {3} ^ {*}}\right)\right) \\ \left. \beta_ {4} ^ {*} = \frac {1}{2} \left(\mathcal {R} _ {\rho_ {X}} \left(q _ {4} ^ {*} + \sqrt {q _ {1} ^ {*} q _ {3} ^ {*}}\right) + \mathcal {R} _ {\rho_ {X}} \left(q _ {4} ^ {*} - \sqrt {q _ {1} ^ {*} q _ {3} ^ {*}}\right)\right) \right. \\ q _ {1} ^ {*} = \mathcal {G} _ {\bar {\mu} _ {S}} (z) \\ q _ {2} ^ {*} = \alpha \mathcal {G} _ {\bar {\mu} _ {S}} (z) + (1 - \alpha) \frac {1}{z} \\ q _ {3} ^ {*} = \frac {(z - \beta_ {1} ^ {*}) ^ {2}}{\beta_ {4} ^ {* 2}} \mathcal {G} _ {\bar {\mu} _ {S}} (z) - \frac {z - \beta_ {1} ^ {*}}{\beta_ {4} ^ {* 2}} \\ q _ {4} ^ {*} = \frac {z - \beta_ {1} ^ {*}}{\beta_ {4} ^ {*}} \mathcal {G} _ {\bar {\mu} _ {S}} (z) - \frac {1}{\beta_ {4} ^ {*}} \end{array} \right. \tag {76}
$$
Remark 5. The simplifications in (75) are derived with the assumption that  $\beta_4^* \neq 0$ . However, in the initial set of equations (71), if  $\rho_X$  is symmetric measure then  $\beta_4^* = q_4^* = 0$  is a solution. If  $\rho_X$  is symmetric, then  $\mathcal{R}_{\rho_X}(-z) = -\mathcal{R}_{\rho_X}(z)$ , and plugging  $q_4^* = 0$  in the expression for  $\beta_4^*$  in (71), we find that  $\beta_4^* = 0$ .
D.3 Overlaps and the optimal singular values
From (59), (72), we find:
$$
\begin{array}{l} O _ {Y} (\gamma , \sigma_ {i}) \approx \frac {1}{\pi \bar {\mu} _ {S} (\gamma)} \operatorname {I m} \lim  _ {z \rightarrow \gamma - \mathrm {i} 0 ^ {+}} \frac {\beta_ {4} ^ {*}}{Z _ {2} (z)} \boldsymbol {y} _ {i} ^ {(r) \top} \boldsymbol {G} _ {Y \top Y} \left(\frac {Z _ {1} (z)}{Z _ {2} (z)}\right) \boldsymbol {Y} ^ {\top} \boldsymbol {y} _ {i} ^ {(l)} \tag {77} \\ = \frac {1}{\pi \bar {\mu} _ {S} (\gamma)} \operatorname {I m} \lim  _ {z \rightarrow \gamma - \mathrm {i} 0 ^ {+}} \beta_ {4} ^ {*} \frac {\sigma_ {i}}{Z _ {1} (z) - Z _ {2} (z) \sigma_ {i} ^ {2}} \\ \end{array}
$$
From the overlap, we can compute the optimal singular values:
$$
\begin{array}{l} \widehat {\xi} _ {y _ {i}} ^ {*} \approx \frac {1}{N} \sum_ {j = 1} ^ {N} \sigma_ {j} O _ {Y} (\gamma_ {i}, \sigma_ {j}) \\ \approx \frac {1}{\pi \bar {\mu} _ {S} (\gamma_ {i})} \mathrm {I m} \lim _ {z \to \gamma_ {i} - \mathrm {i 0} ^ {+}} \frac {1}{N} \sum_ {j = 1} ^ {N} \beta_ {4} ^ {*} \frac {\sigma_ {j} ^ {2}}{Z _ {1} (z) - Z _ {2} (z) \sigma_ {j} ^ {2}} \\ = \frac {1}{\pi \bar {\mu} _ {S} (\gamma_ {i})} \operatorname {I m} \lim  _ {z \rightarrow \gamma_ {i} - \mathrm {i} 0 ^ {+}} \frac {1}{N} \frac {\beta_ {4} ^ {*}}{Z _ {2} (z)} \sum_ {j = 1} ^ {N} \frac {\sigma_ {j} ^ {2}}{\frac {Z _ {1} (z)}{Z _ {2} (z)} - \sigma_ {j} ^ {2}} \tag {78} \\ = \frac {1}{\pi \bar {\mu} _ {S} (\gamma_ {i})} \operatorname {I m} \lim  _ {z \rightarrow \gamma_ {i} - \mathrm {i} 0 ^ {+}} \frac {1}{N} \frac {\beta_ {4} ^ {*}}{Z _ {2} (z)} \sum_ {j = 1} ^ {N} \left[ \frac {\frac {Z _ {1} (z)}{Z _ {2} (z)}}{\frac {Z _ {1} (z)}{Z _ {2} (z)} - \sigma_ {j} ^ {2}} - 1 \right] \\ \approx \frac {1}{\pi \bar {\mu} _ {S} (\gamma_ {i})} \mathrm {I m} \lim _ {z \to \gamma_ {i} - \mathrm {i 0} ^ {+}} \frac {\beta_ {4} ^ {*} Z _ {1} (z)}{Z _ {2} (z) ^ {2}} \mathcal {G} _ {\rho_ {Y}} \big (\frac {Z _ {1} (z)}{Z _ {2} (z)} \big) - \frac {\beta_ {4} ^ {*}}{Z _ {2} (z)} \\ = \frac {1}{\pi \bar {\mu} _ {S} (\gamma_ {i})} \operatorname {I m} \lim  _ {z \rightarrow \gamma_ {i} - \mathrm {i} 0 ^ {+}} q _ {4} ^ {*} \\ \end{array}
$$
where in the last equality we used the solution we have found in (71). Note that, based on (76), we do not need to have any knowledge about  $\rho_{Y}$  to compute  $q_4^*$ . In the end, we need to divide the estimator by  $\sqrt{\kappa}$  as we have absorbed it into  $\pmb{Y}$ .
D.3.1 Recovering the rectangular RIE for a denoising problem
Note that if in the model (60), we put  $\mathbf{X} = \mathbf{I}$  the model reduces to the additive denoising of  $\mathbf{Y}$ , and we recover the estimator recently proposed in [19] for the rectangular case.
For  $\mathbf{X} = \mathbf{I}$ ,  $\mathcal{R}_{\rho_X}(z) = 1$ , so (76) reduces to:
$$
\left\{ \begin{array}{l} \beta_ {1} ^ {*} = \frac {\mathcal {C} _ {\mu_ {W}} ^ {(\alpha)} \left(q _ {1} ^ {*} q _ {2} ^ {*}\right)}{q _ {1} ^ {*}}, \quad \beta_ {2} ^ {*} = \alpha \frac {\mathcal {C} _ {\mu_ {W}} ^ {(\alpha)} \left(q _ {1} ^ {*} q _ {2} ^ {*}\right)}{q _ {2} ^ {*}}, \quad \beta_ {3} ^ {*} = 0, \quad \beta_ {4} ^ {*} = 1 \\ q _ {1} ^ {*} = \mathcal {G} _ {\bar {\mu} _ {S}} (z), \quad q _ {2} ^ {*} = \alpha \mathcal {G} _ {\bar {\mu} _ {S}} (z) + (1 - \alpha) \frac {1}{z} \\ q _ {3} ^ {*} = (z - \beta_ {1} ^ {*}) ^ {2} \mathcal {G} _ {\bar {\mu} _ {S}} (z) - (z - \beta_ {1} ^ {*}), \quad q _ {4} ^ {*} = (z - \beta_ {1} ^ {*}) \mathcal {G} _ {\bar {\mu} _ {S}} (z) - 1 \end{array} \right. \tag {79}
$$
From (78), we have:
$$
\begin{array}{l} \widehat {\xi} _ {y _ {i}} ^ {*} = \frac {1}{\pi \bar {\mu} _ {S} (\gamma_ {i})} \operatorname {I m} \lim  _ {z \rightarrow \gamma_ {i} - \mathrm {i} 0 ^ {+}} q _ {4} ^ {*} \\ = \frac {1}{\pi \bar {\mu} _ {S} (\gamma_ {i})} \operatorname {I m} \lim  _ {z \rightarrow \gamma_ {i} - \mathrm {i} 0 ^ {+}} z \mathcal {G} _ {\bar {\mu} _ {S}} (z) - \beta_ {1} ^ {*} \mathcal {G} _ {\bar {\mu} _ {S}} (z) - 1 \\ = \frac {1}{\pi \bar {\mu} _ {S} (\gamma_ {i})} \operatorname {I m} \lim  _ {z \rightarrow \gamma_ {i} - \mathrm {i} 0 ^ {+}} z \mathcal {G} _ {\bar {\mu} _ {S}} (z) - \frac {\mathcal {C} _ {\mu_ {W}} ^ {(\alpha)} (q _ {1} ^ {*} q _ {2} ^ {*})}{q _ {1} ^ {*}} \mathcal {G} _ {\bar {\mu} _ {S}} (z) - 1 \\ = \frac {1}{\pi \bar {\mu} _ {S} (\gamma_ {i})} \operatorname {I m} \lim  _ {z \rightarrow \gamma_ {i} - \mathrm {i} 0 ^ {+}} z \mathcal {G} _ {\bar {\mu} _ {S}} (z) - \mathcal {C} _ {\mu_ {W}} ^ {(\alpha)} \left(q _ {1} ^ {*} q _ {2} ^ {*}\right) - 1 \\ = \frac {1}{\pi \bar {\mu} _ {S} (\gamma_ {i})} \operatorname {I m} \lim  _ {z \rightarrow \gamma_ {i} - \mathrm {i} 0 ^ {+}} z \mathcal {G} _ {\bar {\mu} _ {S}} (z) - \mathcal {C} _ {\mu_ {W}} ^ {(\alpha)} \left(\mathcal {G} _ {\bar {\mu} _ {S}} (z) \left(\alpha \mathcal {G} _ {\bar {\mu} _ {S}} (z) + (1 - \alpha) \frac {1}{z}\right)\right) - 1 \\ \stackrel {\mathrm {(a)}} {=} \frac {1}{\pi \bar {\mu} _ {S} (\gamma_ {i})} \mathrm {I m} \left[ \gamma_ {i} \mathcal {G} _ {\bar {\mu} _ {S}} (\gamma_ {i} - \mathrm {i} 0 ^ {+}) - \mathcal {C} _ {\mu_ {W}} ^ {(\alpha)} \bigg (\frac {1}{\gamma_ {i}} \mathcal {G} _ {\bar {\mu} _ {S}} (\gamma_ {i} - \mathrm {i} 0 ^ {+}) \Big (1 - \alpha + \alpha \gamma_ {i} \mathcal {G} _ {\bar {\mu} _ {S}} (\gamma_ {i} - \mathrm {i} 0 ^ {+}) \Big) \bigg) \right] \\ \stackrel {(b)} {=} \gamma_ {i} - \frac {1}{\pi \bar {\mu} _ {S} (\gamma_ {i})} \operatorname {I m} \mathcal {C} _ {\mu_ {W}} ^ {(\alpha)} \left(\frac {1 - \alpha}{\gamma_ {i}} \pi \mathsf {H} [ \bar {\mu} _ {S} ] (\gamma_ {i}) + \alpha (\pi \mathsf {H} [ \bar {\mu} _ {S} ] (\gamma_ {i})) ^ {2} - \alpha (\pi \bar {\mu} _ {S} (\gamma_ {i})) ^ {2} \right. \\ \left. + \mathrm {i} \pi \bar {\mu} _ {S} (\gamma_ {i}) \left(\frac {1 - \alpha}{\gamma_ {i}} + 2 \alpha \pi \mathsf {H} [ \bar {\mu} _ {S} ] (\gamma_ {i})\right)\right) \tag {80} \\ \end{array}
$$
where in (a) we used the analyticity of rectangular R-transform [57], and in (b), we used Plemelj formula (6). Note that, the final estimator should be divided by the  $\sqrt{\kappa}$ .
D.4 Examples