SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[48] Christopher M Bishop. Variational principal components. 1999.
[49] Yew Jin Lim and Yee Whye Teh. Variational Bayesian approach to movie rating prediction. In Proceedings of KDD cup and workshop, volume 7, pages 15-21, 2007.
[50] Alexander Ilin and Tapani Raiko. Practical approaches to principal component analysis in the presence of missing values. The Journal of Machine Learning Research, 11:1957-2000, 2010.
[51] Shinichi Nakajima, Masashi Sugiyama, S Derin Babacan, and Ryota Tomioka. Global analytic solution of fully-observed variational bayesian matrix factorization. The Journal of Machine Learning Research, 14(1):1-37, 2013.
[52] Salma Tarmoun, Guilherme Franca, Benjamin D Haeffele, and Rene Vidal. Understanding the dynamics of gradient flow in overparameterized linear models. In International Conference on Machine Learning, pages 10153-10161. PMLR, 2021.
[53] Antoine Bodin and Nicolas Macris. Gradient flow on extensive-rank positive semi-definite matrix denoising. arXiv preprint arXiv:2303.09474, 2023.
[54] Daniel D Lee and H Sebastian Seung. Learning the parts of objects by non-negative matrix factorization. Nature, 401(6755):788-791, 1999.
[55] Marco Mondelli and Ramji Venkataramanan. Approximate message passing with spectral initialization for generalized linear models. In International Conference on Artificial Intelligence and Statistics, pages 397-405. PMLR, 2021.
[56] James A Mingo and Roland Speicher. Free probability and random matrices, volume 35. Springer, 2017.
[57] Florent Benaych-Georges. Rectangular random matrices, related convolution. Probability Theory and Related Fields, 144:471-515, 2009.
[58] Marc Potters and Jean-Philippe Bouchaud. A First Course in Random Matrix Theory: For Physicists, Engineers and Data Scientists. Cambridge University Press, 2020.
[59] Roger A Horn and Charles R Johnson. Matrix analysis. Cambridge university press, 2012.
[60] Jean Zinn-Justin. Quantum field theory and critical phenomena, volume 171. Oxford university press, 2021.
[61] Dennis S Bernstein. Matrix mathematics. In Matrix Mathematics. Princeton university press, 2009.
[62] Harish-Chandra. Differential operators on a semisimple lie algebra. American Journal of Mathematics, pages 87-120, 1957.
[63] Alice Guionnet and Jonathan Husson. Asymptotics of k dimensional spherical integrals and applications. ALEA, 19:769-797, 2022.
A Posterior mean estimator is in the RIE class
In this section, we show that for rotational invariant priors, the posterior mean estimator is inside the RIE class. For each of the estimators of  $\mathbf{X}, \mathbf{Y}$ , we present an equivalent definition of the RIE, then we show that posterior mean estimator satisfies this definition.
A.1 X Estimator
Lemma 1. Given the observation matrix  $S$ , let  $\hat{\mathbf{X}}(S)$  be an estimator of  $\mathbf{X}$ . Then  $\hat{\mathbf{X}}(S)$  is a RIE if and only if for any orthogonal matrices  $\mathbf{U} \in \mathbb{R}^{N \times N}$ ,  $\mathbf{V} \in \mathbb{R}^{M \times M}$ :
$$
\hat {\boldsymbol {X}} (\boldsymbol {U} \boldsymbol {S} \boldsymbol {V} ^ {\intercal}) = \boldsymbol {U} \hat {\boldsymbol {X}} (\boldsymbol {S}) \boldsymbol {U} ^ {\intercal} \tag {22}
$$
Proof. If  $\hat{\pmb{X}}(S)$  is a RIE, then the property (22) clearly follows from the definition (2). Now we turn to the converse.
Suppose that an estimator  $\hat{\pmb{X}} (\pmb {S})$  satisfies (22). First, we show that if the observation matrix is diagonal, then the estimator is also diagonal. Consider the observation matrix to be  $S^{\mathrm{diag}} = \left[\operatorname {diag}(\mathrm{s}_1,\dots ,\mathrm{s_N})\mid \mathbf{0}_{N\times (M - N)}\right]$  . Let  $\pmb{I}_k^{-}\in \mathbb{R}^{N\times N},\pmb{J}_k^{-}\in \mathbb{R}^{M\times M}$  be diagonal matrices with diagonal entries all one except the  $k$  -th entry which is  $-1$  . Note that for  $1\leq k\leq N$  , we have  $S^{\mathrm{diag}} = I_k^- S^{\mathrm{diag}}J_k^-$  . Moreover, matrices  $\pmb{I}_k^{-},\pmb{J}_k^{-}$  are indeed orthogonal. For any  $1\leq k\leq N$  , from the property we have:
$$
\hat {\boldsymbol {X}} \left(\boldsymbol {S} ^ {\mathrm {d i a g}}\right) = \hat {\boldsymbol {X}} \left(\boldsymbol {I} _ {k} ^ {-} \boldsymbol {S} ^ {\mathrm {d i a g}} \boldsymbol {J} _ {k} ^ {-}\right) = \boldsymbol {I} _ {k} ^ {-} \hat {\boldsymbol {X}} \left(\boldsymbol {S} ^ {\mathrm {d i a g}}\right) \boldsymbol {I} _ {k} ^ {-} \tag {23}
$$
This implies that all entries on the  $k$ -th row and  $k$ -th column of  $\hat{X}(S^{\mathrm{diag}})$  are zero except the  $k$ -th entry on the diagonal. Since this holds for any  $k$ , we conclude that  $\hat{X}(S^{\mathrm{diag}})$  is diagonal.
Now, for a given general observation matrix with SVD  $S = U_S \Gamma V_S^\top$ , put  $U = U_S^\top$ ,  $V = V_S^\top$  in the property (22). We have:
$$
\hat {\boldsymbol {X}} (\boldsymbol {\Gamma}) = \boldsymbol {U} _ {S} ^ {\intercal} \hat {\boldsymbol {X}} (\boldsymbol {S}) \boldsymbol {U} _ {S}
$$
From the argument above, the matrix on the lhs is diagonal. Consequently, the matrix  $\mathbf{U}_S^\top \hat{\mathbf{X}}(\mathbf{S})\mathbf{U}_S$  is diagonal which implies that the columns of  $\mathbf{U}_S$  are eigenvectors of  $\hat{\mathbf{X}}(\mathbf{S})$ . Therefore,  $\hat{\mathbf{X}}(\mathbf{S})$  is a RIE.
Now, we prove that the posterior mean estimator  $\hat{\pmb{X}}^{*}(\pmb {S}) = \mathbb{E}[\pmb {X}|\pmb {S}]$  has the property (22), and therefore belongs to the RIE class. For simplicity, we drop the SNR factor  $\sqrt{\kappa}$ . For any orthogonal matrices  $\pmb {U}\in \mathbb{R}^{N\times N}$ $\pmb {V}\in \mathbb{R}^{M\times M}$ , we have:
$$
\begin{array}{l} \mathbb {E} [ \boldsymbol {X} | \boldsymbol {U} \boldsymbol {S} \boldsymbol {V} ^ {\intercal} ] = \frac {\int d \boldsymbol {Y} d \tilde {\boldsymbol {X}} \tilde {\boldsymbol {X}} P _ {X} (\tilde {\boldsymbol {X}}) P _ {Y} (\boldsymbol {Y}) P _ {W} (\boldsymbol {U} \boldsymbol {S} \boldsymbol {V} ^ {\intercal} - \tilde {\boldsymbol {X}} \boldsymbol {Y})}{\int d \boldsymbol {Y} d \tilde {\boldsymbol {X}} P _ {X} (\tilde {\boldsymbol {X}}) P _ {Y} (\boldsymbol {Y}) P _ {W} (\boldsymbol {U} \boldsymbol {S} \boldsymbol {V} ^ {\intercal} - \tilde {\boldsymbol {X}} \boldsymbol {Y})} \\ \stackrel {\mathrm {(a)}} {=} \frac {\int d \boldsymbol {Y} d \tilde {\boldsymbol {X}} \boldsymbol {U} \tilde {\boldsymbol {X}} \boldsymbol {U} ^ {\intercal} P _ {X} (\tilde {\boldsymbol {X}}) P _ {Y} (\boldsymbol {Y}) P _ {W} (\boldsymbol {U} \boldsymbol {S} \boldsymbol {V} ^ {\intercal} - \boldsymbol {U} \tilde {\boldsymbol {X}} \boldsymbol {U} ^ {\intercal} \boldsymbol {Y})}{\int d \boldsymbol {Y} d \tilde {\boldsymbol {X}} P _ {X} (\tilde {\boldsymbol {X}}) P _ {Y} (\boldsymbol {Y}) P _ {W} (\boldsymbol {U} \boldsymbol {S} \boldsymbol {V} ^ {\intercal} - \boldsymbol {U} \tilde {\boldsymbol {X}} \boldsymbol {U} ^ {\intercal} \boldsymbol {Y})} \\ \stackrel {\mathrm {(b)}} {=} \frac {\int d \boldsymbol {Y} d \tilde {\boldsymbol {X}} \boldsymbol {U} \tilde {\boldsymbol {X}} \boldsymbol {U} ^ {\intercal} P _ {X} (\tilde {\boldsymbol {X}}) P _ {Y} (\boldsymbol {Y}) P _ {W} (\boldsymbol {U} \boldsymbol {S} \boldsymbol {V} ^ {\intercal} - \boldsymbol {U} \tilde {\boldsymbol {X}} \boldsymbol {U} ^ {\intercal} \boldsymbol {U} \boldsymbol {Y} \boldsymbol {V} ^ {\intercal})}{\int d \boldsymbol {Y} d \tilde {\boldsymbol {X}} P _ {X} (\tilde {\boldsymbol {X}}) P _ {Y} (\boldsymbol {Y}) P _ {W} (\boldsymbol {U} \boldsymbol {S} \boldsymbol {V} ^ {\intercal} - \boldsymbol {U} \tilde {\boldsymbol {X}} \boldsymbol {U} ^ {\intercal} \boldsymbol {U} \boldsymbol {Y} \boldsymbol {V} {} ^ {\intercal})} \\ \stackrel {\mathrm {(c)}} {=} \boldsymbol {U} \left\{\frac {\int d \boldsymbol {Y} d \tilde {\boldsymbol {X}} \tilde {\boldsymbol {X}} P _ {X} (\tilde {\boldsymbol {X}}) P _ {Y} (\boldsymbol {Y}) P _ {W} (\boldsymbol {S} - \tilde {\boldsymbol {X}} \boldsymbol {Y})}{\int d \boldsymbol {Y} d \tilde {\boldsymbol {X}} P _ {X} (\tilde {\boldsymbol {X}}) P _ {Y} (\boldsymbol {Y}) P _ {W} (\boldsymbol {S} - \tilde {\boldsymbol {X}} \boldsymbol {Y})} \right\} \boldsymbol {U} ^ {\intercal} \\ = \boldsymbol {U} \mathbb {E} [ \boldsymbol {X} | \boldsymbol {S} ] \boldsymbol {U} ^ {\intercal} \\ \end{array}
$$
where in (a), we changed variables  $\tilde{\pmb{X}}\rightarrow \pmb {U}\tilde{\pmb{X}}\pmb{U}^{\intercal}$  , used  $|\operatorname *{det}\boldsymbol {U}| = 1$  , and rotational invariance of  $P_{X}$ $P_{X}(\tilde{\pmb{X}}) = P_{X}(\pmb {U}\tilde{\pmb{X}}\pmb{U}^{\intercal})$  . In (b), we changed variables  $\pmb {Y}\rightarrow \pmb {U}\pmb {Y}\pmb{V}^{\intercal}$  , used  $|\operatorname *{det}\boldsymbol {U}| = |\operatorname *{det}\boldsymbol {V}| = 1$  and bi-rotational invariance of  $P_{Y}$ $P_{Y}(\pmb {Y}) = P_{Y}(\pmb {U}\pmb {Y}\pmb{V}^{\intercal})$  . In (c), we used the bi-rotational invariance property of  $P_W$  , namely  $P_W(\pmb {USV}^\intercal -\pmb {UX}\pmb {Y}\pmb{V}^\intercal) = P_W(\pmb {S} - \tilde{\pmb{X}}\pmb {Y})$
A.2 Y Estimator
Lemma 2. Given the observation matrix  $S$ , let  $\hat{\mathbf{Y}}(S)$  be an estimator for  $\mathbf{Y}$ . Then  $\hat{\mathbf{Y}}(S)$  is a RIE if and only if for any orthogonal matrices  $\mathbf{U} \in \mathbb{R}^{N \times N}$ ,  $\mathbf{V} \in \mathbb{R}^{M \times M}$ :
$$
\hat {\mathbf {Y}} (\mathbf {U} \mathbf {S} \mathbf {V} ^ {\intercal}) = \mathbf {U} \hat {\mathbf {Y}} (\mathbf {S}) \mathbf {V} ^ {\intercal} \tag {24}
$$
Proof. If  $\hat{Y}(S)$  is a RIE, then this property clearly follows from the definition (4). Let us now show the converse.
Suppose that an estimator  $\hat{\pmb{Y}} (\pmb {S})$  satisfies (24). First, we show that if the observation matrix is diagonal, then the estimator is also diagonal. Consider the observation matrix to be  $S^{\mathrm{diag}} = \left[\mathrm{diag}(\mathrm{s}_1,\dots ,\mathrm{s_N})\mid \mathbf{0}_{N\times (M - N)}\right]$  . Let  $\pmb{I}_k^{-}\in \mathbb{R}^{N\times N},\pmb{J}_k^{-}\in \mathbb{R}^{M\times M}$  be diagonal matrices with diagonal entries all one except the  $k$  -th entry which is  $-1$  . Note that for  $1\leq k\leq N$  , we have  $S^{\mathrm{diag}} = I_k^- S^{\mathrm{diag}}J_k^{-}$  . Moreover, matrices  $\pmb{I}_k^{-},\pmb{J}_k^{-}$  are indeed orthogonal. For any  $1\leq k\leq N$  , from the property we have:
$$
\hat {\boldsymbol {Y}} (\boldsymbol {S} ^ {\mathrm {d i a g}}) = \hat {\boldsymbol {Y}} \left(\boldsymbol {I} _ {k} ^ {-} \boldsymbol {S} ^ {\mathrm {d i a g}} \boldsymbol {J} _ {k} ^ {-}\right) = \boldsymbol {I} _ {k} ^ {-} \hat {\boldsymbol {Y}} \left(\boldsymbol {S} ^ {\mathrm {d i a g}}\right) \boldsymbol {J} _ {k} ^ {-} \tag {25}
$$
This implies that all entries on the  $k$ -th row and  $k$ -th column of  $\hat{Y}(S^{\mathrm{diag}})$  is zero except the  $k$ -th entry on the diagonal. Since this holds for any  $k$ , we conclude that  $\hat{Y}(S^{\mathrm{diag}})$  is diagonal.
Now, for a given general observation matrix  $S = \mathbf{U}_S\Gamma \mathbf{V}_S^\top$ , put  $\mathbf{U} = \mathbf{U}_S^\top$ ,  $\mathbf{V} = \mathbf{V}_S^\top$  in the property (24). We have:
$$
\hat {\mathbf {Y}} (\boldsymbol {\Gamma}) = \boldsymbol {U} _ {S} ^ {\intercal} \hat {\mathbf {Y}} (\boldsymbol {S}) \boldsymbol {V} _ {S}
$$
From the argument above, the matrix on the lhs is diagonal. Consequently, the matrix  $\mathbf{U}_S^\top \hat{\mathbf{Y}}(\mathbf{S})\mathbf{V}_S$  is diagonal which implies that the columns of  $\mathbf{U}_S$ ,  $\mathbf{V}_S$  are the left and right singular vectors of  $\hat{\mathbf{Y}}(\mathbf{S})$ . Therefore,  $\hat{\mathbf{Y}}(\mathbf{S})$  is a RIE.
Now, we prove that the posterior mean estimator  $\hat{\mathbf{Y}}^{*}(\mathbf{S}) = \mathbb{E}[\mathbf{Y}|\mathbf{S}]$  has the property (24), and it is inside the RIE class. For simplicity, we drop the SNR factor  $\sqrt{\kappa}$ . For any orthogonal matrices  $\mathbf{U} \in \mathbb{R}^{N \times N}$ ,  $\mathbf{V} \in \mathbb{R}^{M \times M}$ , we have:
$$
\begin{array}{l} \mathbb {E} [ \boldsymbol {Y} | \boldsymbol {U} \boldsymbol {S} \boldsymbol {V} ^ {\intercal} ] = \frac {\int d \boldsymbol {X} d \tilde {\boldsymbol {Y}} \tilde {\boldsymbol {Y}} P _ {X} (\boldsymbol {X}) P _ {Y} (\tilde {\boldsymbol {Y}}) P _ {W} (\boldsymbol {U} \boldsymbol {S} \boldsymbol {V} ^ {\intercal} - \boldsymbol {X} \tilde {\boldsymbol {Y}})}{\int d \boldsymbol {X} d \tilde {\boldsymbol {Y}} P _ {X} (\boldsymbol {X}) P _ {Y} (\tilde {\boldsymbol {Y}}) P _ {W} (\boldsymbol {U} \boldsymbol {S} \boldsymbol {V} ^ {\intercal} - \boldsymbol {X} \tilde {\boldsymbol {Y}})} \\ \stackrel {\mathrm {(a)}} {=} \frac {\int d \boldsymbol {X} d \tilde {\boldsymbol {Y}} \boldsymbol {U} \tilde {\boldsymbol {Y}} \boldsymbol {V} ^ {\intercal} P _ {X} (\boldsymbol {X}) P _ {Y} (\tilde {\boldsymbol {Y}}) P _ {W} (\boldsymbol {U} \boldsymbol {S} \boldsymbol {V} ^ {\intercal} - \boldsymbol {X} \boldsymbol {U} \tilde {\boldsymbol {Y}} \boldsymbol {V} ^ {\intercal})}{\int d \boldsymbol {X} d \tilde {\boldsymbol {Y}} P _ {X} (\boldsymbol {X}) P _ {Y} (\tilde {\boldsymbol {Y}}) P _ {W} (\boldsymbol {U} \boldsymbol {S} \boldsymbol {V} ^ {\intercal} - \boldsymbol {X} \boldsymbol {U} \tilde {\boldsymbol {Y}} \boldsymbol {V} ^ {\intercal})} \\ \stackrel {\mathrm {(b)}} {=} \frac {\int d \boldsymbol {X} d \tilde {\boldsymbol {Y}} \boldsymbol {U} \tilde {\boldsymbol {Y}} \boldsymbol {V} ^ {\intercal} P _ {X} (\boldsymbol {X}) P _ {Y} (\tilde {\boldsymbol {Y}}) P _ {W} (\boldsymbol {U} \boldsymbol {S} \boldsymbol {V} ^ {\intercal} - \boldsymbol {U} \boldsymbol {X} \boldsymbol {U} ^ {\intercal} \boldsymbol {U} \tilde {\boldsymbol {Y}} \boldsymbol {V} ^ {\intercal})}{\int d \boldsymbol {X} d \tilde {\boldsymbol {Y}} P _ {X} (\boldsymbol {X}) P _ {Y} (\tilde {\boldsymbol {Y}}) P _ {W} (\boldsymbol {U} \boldsymbol {S} \boldsymbol {V} ^ {\intercal} - \boldsymbol {U} \boldsymbol {X} \boldsymbol {U} ^ {\intercal} \boldsymbol {U} \tilde {\boldsymbol {Y}} \boldsymbol {V} {} ^ {\intercal})} \\ \stackrel {\mathrm {(c)}} {=} \boldsymbol {U} \left\{\frac {\int d \boldsymbol {X} d \tilde {\boldsymbol {Y}} \tilde {\boldsymbol {Y}} P _ {X} (\boldsymbol {X}) P _ {Y} (\tilde {\boldsymbol {Y}}) P _ {W} (\boldsymbol {S} - \boldsymbol {X} \tilde {\boldsymbol {Y}})}{\int d \boldsymbol {X} d \tilde {\boldsymbol {Y}} P _ {X} (\boldsymbol {X}) P _ {Y} (\tilde {\boldsymbol {Y}}) P _ {W} (\boldsymbol {S} - \boldsymbol {X} \tilde {\boldsymbol {Y}})} \right\} \boldsymbol {V} ^ {\intercal} \\ = \boldsymbol {U} \mathbb {E} [ \boldsymbol {Y} | \boldsymbol {S} ] \boldsymbol {V} ^ {\intercal} \\ \end{array}
$$
where in (a), we changed variables  $\tilde{\pmb{Y}}\rightarrow \pmb {U}\tilde{\pmb{Y}}\pmb{V}^{\intercal}$  , used  $|\operatorname *{det}\boldsymbol {U}| = |\operatorname *{det}\boldsymbol {V}| = 1$  , and bi-rotational invariance of  $P_{Y}$ $P_{Y}(\tilde{\pmb{Y}}) = P_{Y}(\pmb {U}\tilde{\pmb{Y}}\pmb{V}^{\intercal})$  . In (b), we changed variables  $\pmb {X}\rightarrow \pmb {UXU}^{\intercal}$  , used  $|\operatorname *{det}\boldsymbol {U}| = 1$  , and rotational invariance of  $P_{X}$ $P_{X}(\pmb {X}) = P_{X}(\pmb {U}\pmb {X}\pmb{U}^{\intercal})$  . In (c), we used the bi-rotational invariance property of  $P_W$  , namely  $P_W(USV^\intercal -UX\tilde{Y} V^\intercal) = P_W(S - X\tilde{Y})$
B The replica method for deriving the resolvent relation
In this section we present the replica method used to obtain the resolvent relation. For simplicity of notation we use  $G(z) \equiv G_S(z)$  for the resolvent of a random matrix  $S$ .
First, we express the entries of the resolvent  $G(z)$  using the Gaussian integral representation of an inverse matrix [60]:
$$
\begin{array}{l} G _ {i j} (z) = \sqrt {\frac {1}{(2 \pi) ^ {N + M} \det (z \pmb {I} - \pmb {S})}} \int \Big (\prod_ {k = 1} ^ {M + N} d \eta_ {k} \Big) \eta_ {i} \eta_ {j} \exp \Big \{- \frac {1}{2} \pmb {\eta} ^ {\intercal} (z \pmb {I} - \pmb {S}) \pmb {\eta} \Big \} \\ = \frac {\int \left(\prod_ {k = 1} ^ {M + N} d \eta_ {k}\right) \eta_ {i} \eta_ {j} \exp \left\{- \frac {1}{2} \boldsymbol {\eta} ^ {\intercal} (z \boldsymbol {I} - \boldsymbol {\mathcal {S}}) \boldsymbol {\eta} \right\}}{\int \left(\prod_ {k = 1} ^ {M + N} d \eta_ {k}\right) \exp \left\{- \frac {1}{2} \boldsymbol {\eta} ^ {\intercal} (z \boldsymbol {I} - \boldsymbol {\mathcal {S}}) \boldsymbol {\eta} \right\}} \tag {26} \\ \end{array}
$$
For  $z$  not close to the real axis, the resolvent is expected to exhibit self-averaging behavior in the limit of large  $\mathbf{N}$ , meaning that it will not depend on the particular matrix realization. Thus, we can examine the resolvent  $G_{S}(z)$  by analyzing its ensemble average, denoted by  $\langle .\rangle$  in the following.
$$
\left\langle G _ {i j} (z) \right\rangle = \left\langle \frac {1}{\mathcal {Z}} \int \left(\prod_ {k = 1} ^ {M + N} d \eta_ {k}\right) \eta_ {i} \eta_ {j} \exp \left\{- \frac {1}{2} \boldsymbol {\eta} ^ {\intercal} (z \boldsymbol {I} - \boldsymbol {S}) \boldsymbol {\eta} \right\} \right\rangle \tag {27}
$$
where  $\mathcal{Z}$  is the denominator in (26). Computing the average is, in general, non-trivial. However, the replica method provides us with a technique to overcome this issue by employing the following identity:
$$
\begin{array}{l} \left\langle G _ {i j} (z) \right\rangle = \lim  _ {n \rightarrow 0} \left\langle \mathcal {Z} ^ {n - 1} \int \left(\prod_ {k = 1} ^ {M + N} d \eta_ {k}\right) \eta_ {i} \eta_ {j} \exp \left\{- \frac {1}{2} \boldsymbol {\eta} ^ {\intercal} (z \boldsymbol {I} - \boldsymbol {\mathcal {S}}) \boldsymbol {\eta} \right\}\right\rangle \tag {28} \\ = \lim  _ {n \rightarrow 0} \left\langle \int \left(\prod_ {k = 1} ^ {M + N} \prod_ {\tau = 1} ^ {n} d \eta_ {k} ^ {(\tau)}\right) \eta_ {i} ^ {(1)} \eta_ {j} ^ {(1)} \exp \left\{- \frac {1}{2} \sum_ {\tau = 1} ^ {n} \boldsymbol {\eta} ^ {(\tau) \intercal} (z \boldsymbol {I} - \boldsymbol {\mathcal {S}}) \boldsymbol {\eta} ^ {(\tau)} \right\}\right\rangle \\ \end{array}
$$
So, the problem now is reduced to computation of an average over  $n$  copies (or replicas) of the initial system (26). After computing the average value (the bracket) in (28), we can perform an analytical continuation of the result to real values of  $n$  and then take the limit  $n \to 0$ . Throughout, we assume as is common in the replica method, that the analytical continuation can be done with only  $n$  different sets of points. Of course, this is a totally uncontrolled step that comes with no guarantees.
C Derivation of the RIE for  $X$
In this section, we consider estimating  $X$ , and treat both  $Y$  and  $W$  as noise. We consider  $X$  to be fixed, and the observation model:
$$
\boldsymbol {S} = \boldsymbol {X} \boldsymbol {U} _ {1} \boldsymbol {Y} \boldsymbol {V} _ {1} ^ {\intercal} + \boldsymbol {U} _ {2} \boldsymbol {W} \boldsymbol {V} _ {2} ^ {\intercal} \tag {29}
$$
where  $\mathbf{Y},\mathbf{W}\in \mathbb{R}^{N\times M}$  are fixed matrices with limiting singular value distribution  $\mu_Y,\mu_W$  , and  $\pmb {U}_1,\pmb {U}_2\in \mathbb{R}^{N\times N}$ $V_{1},V_{2}\in \mathbb{R}^{M\times M}$  are independent random Haar matrices.
Construct the hermitization  $\mathcal{S} \in \mathbb{R}^{(N + M) \times (N + M)}$  from  $\mathcal{S}$  as
For simplicity of notation, we use  $\pmb{T} \equiv \pmb{X}\pmb{U}_1\pmb{Y}\pmb{V}_1^\top$ ,  $\pmb{\mathcal{T}} \in \mathbb{R}^{(N + M) \times (N + M)}$  the hermitization of  $\pmb{T}$ , and  $\widetilde{\mathcal{W}}$  denotes the hermitization of the matrix  $U_2\mathbf{W}\mathbf{V}_2^\top$ .
C.1 Resolvent relation
We want to find a relation between  $G(z) \equiv G_{\mathcal{S}}(z)$ , and the signal matrix  $X$ . From (28), we have
$$
\begin{array}{l} \langle G _ {i j} (z) \rangle = \lim  _ {n \rightarrow \infty} \int \Big (\prod_ {k = 1} ^ {N + M} \prod_ {\tau = 1} ^ {n} d \eta_ {k} ^ {(\tau)} \Big) \eta_ {i} ^ {(1)} \eta_ {j} ^ {(1)} \Big \langle \exp \left\{- \frac {1}{2} \sum_ {\tau = 1} ^ {n} \boldsymbol {\eta} ^ {(\tau) \mathsf {T}} (z \boldsymbol {I} - \boldsymbol {\mathcal {S}}) \boldsymbol {\eta} ^ {(\tau)} \right\} \Big \rangle_ {\boldsymbol {U} _ {1}, \boldsymbol {U} _ {2}, \boldsymbol {V} _ {1}, \boldsymbol {V} _ {2}} \\ = \lim  _ {n \rightarrow \infty} \int \left(\prod_ {k = 1} ^ {N + M} \prod_ {\tau = 1} ^ {n} d \eta_ {k} ^ {(\tau)}\right) \eta_ {i} ^ {(1)} \eta_ {j} ^ {(1)} \exp \left\{- \frac {z}{2} \sum_ {\tau = 1} ^ {n} \boldsymbol {\eta} ^ {(\tau) \top} \boldsymbol {\eta} ^ {(\tau)} \right\} \\ \times \left\langle \exp \left\{\frac {1}{2} \sum_ {\tau = 1} ^ {n} \boldsymbol {\eta} ^ {(\tau) ^ {\intercal}} \boldsymbol {\mathcal {T}} \boldsymbol {\eta} ^ {(\tau)} \right\} \right\rangle_ {\boldsymbol {U} _ {1}, \boldsymbol {V} _ {1}} \left\langle \exp \left\{\frac {1}{2} \sum_ {\tau = 1} ^ {n} \boldsymbol {\eta} ^ {(\tau) ^ {\intercal}} \widetilde {\boldsymbol {\mathcal {W}}} \boldsymbol {\eta} ^ {(\tau)} \right\} \right\rangle_ {\boldsymbol {U} _ {2}, \boldsymbol {V} _ {2}} \tag {30} \\ \end{array}
$$
Split each replica  $\pmb{\eta}^{(\tau)}$  into two vectors  $\pmb{a}^{(\tau)} \in \mathbb{R}^N$ ,  $\pmb{b}^{(\tau)} \in \mathbb{R}^M$ ,  $\pmb{\eta}^{(\tau)} = \left[ \begin{array}{c}\pmb{a}^{(\tau)}\\ \pmb{b}^{(\tau)} \end{array} \right]$ . The exponent in the first bracket in (30) can be written as:
$$
\begin{array}{l} \boldsymbol {\eta} ^ {(\tau) \intercal} \boldsymbol {\mathcal {T}} \boldsymbol {\eta} ^ {(\tau)} = \boldsymbol {a} ^ {(\tau) \intercal} \boldsymbol {X} \boldsymbol {U} _ {1} \boldsymbol {Y} \boldsymbol {V} _ {1} ^ {\intercal} \boldsymbol {b} ^ {(\tau)} + \boldsymbol {b} ^ {(\tau) \intercal} \boldsymbol {V} _ {1} \boldsymbol {Y} ^ {\intercal} \boldsymbol {U} _ {1} ^ {\intercal} \boldsymbol {X} \boldsymbol {a} ^ {(\tau)} \\ = 2 \boldsymbol {a} ^ {(\tau) ^ {\intercal}} \boldsymbol {X} \boldsymbol {U} _ {1} \boldsymbol {Y} \boldsymbol {V} _ {1} ^ {\intercal} \boldsymbol {b} ^ {(\tau)} \tag {31} \\ = 2 \operatorname {T r} \boldsymbol {b} ^ {(\tau)} \boldsymbol {a} ^ {(\tau) ^ {\mathsf {T}}} \boldsymbol {X} \boldsymbol {U} _ {1} \boldsymbol {Y} \boldsymbol {V} _ {1} ^ {\mathsf {T}} \\ \end{array}
$$
Using the formula for the rectangular spherical integral [22] (see Theorem 2 in H.1), we find:
$$
\left\langle \exp \left\{\sum_ {\tau = 1} ^ {n} \operatorname {T r} \boldsymbol {b} ^ {(\tau)} \boldsymbol {a} ^ {(\tau) ^ {\mathsf {T}}} \boldsymbol {X} \boldsymbol {U} _ {1} \boldsymbol {Y} \boldsymbol {V} _ {1} ^ {\mathsf {T}} \right\} \right\rangle_ {\boldsymbol {U} _ {1}, \boldsymbol {V} _ {1}} \approx \exp \left\{\frac {N}{2} \sum_ {\tau = 1} ^ {n} \mathcal {Q} _ {\mu_ {Y}} ^ {(\alpha)} \left(\frac {1}{N M} \| \boldsymbol {X} \boldsymbol {a} ^ {(\tau)} \| ^ {2} \| \boldsymbol {b} ^ {(\tau)} \| ^ {2}\right) \right\} \tag {32}
$$
with  $\mathcal{Q}_{\mu_Y}^{(\alpha)}(x) = \int_0^x\frac{\mathcal{C}_{\mu_Y}^{(\alpha)}(t)}{t} dt.$  In (32), we used that  $\pmb{b}^{(\tau)}\pmb{a}^{(\tau)^{\mathrm{T}}}\pmb{X}$  is a rank-one matrix with non-zero singular value  $\| \pmb {b}^{(\tau)}\| \| \pmb {X}\pmb{a}^{(\tau)}\|$ .
Similarly, for the second bracket in (30) we can write:
$$
\begin{array}{l} \boldsymbol {\eta} ^ {(\tau) ^ {\intercal}} \widetilde {\mathcal {W}} \boldsymbol {\eta} ^ {(\tau)} = \boldsymbol {a} ^ {(\tau) ^ {\intercal}} \boldsymbol {U} _ {2} \boldsymbol {W} \boldsymbol {V} _ {2} ^ {\intercal} \boldsymbol {b} ^ {(\tau)} + \boldsymbol {b} ^ {(\tau) ^ {\intercal}} \boldsymbol {V} _ {2} \boldsymbol {W} ^ {\intercal} \boldsymbol {U} _ {2} ^ {\intercal} \boldsymbol {a} ^ {(\tau)} \\ = 2 \boldsymbol {a} ^ {(\tau) \top} \boldsymbol {U} _ {2} \boldsymbol {W} \boldsymbol {V} _ {2} ^ {\top} \boldsymbol {b} ^ {(\tau)} \tag {33} \\ = 2 \operatorname {T r} \boldsymbol {b} ^ {(\tau)} \boldsymbol {a} ^ {(\tau) ^ {\mathsf {T}}} \boldsymbol {U} _ {2} \boldsymbol {W} \boldsymbol {V} _ {2} ^ {\mathsf {T}} \\ \end{array}
$$
which using the formula of rectangular spherical integrals, implies
$$
\left\langle \exp \left\{\sum_ {\tau = 1} ^ {n} \operatorname {T r} \boldsymbol {b} ^ {(\tau)} \boldsymbol {a} ^ {(\tau) ^ {\intercal}} \boldsymbol {U} _ {2} \boldsymbol {W} \boldsymbol {V} _ {2} ^ {\intercal} \right\} \right\rangle_ {\boldsymbol {U} _ {2}, \boldsymbol {V} _ {2}} \approx \exp \left\{\frac {N}{2} \sum_ {\tau = 1} ^ {n} \mathcal {Q} _ {\mu W} ^ {(\alpha)} \left(\frac {1}{N M} \| \boldsymbol {a} ^ {(\tau)} \| ^ {2} \| \boldsymbol {b} ^ {(\tau)} \| ^ {2}\right) \right\} \tag {34}
$$