SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$^{1}\zeta_{1}, \zeta_{3}$  are the only parameters which appear in the final estimator. However, in derivation of the RIE, we have defined other parameters which do not appear in the final estimator and we omit them here.
3.2.1 An estimator for  $X^2$
It is interesting to note that we can construct a RIE for  $X^2$  as  $\widehat{\Xi_{X^2}^*}(S) = \sum_{i=1}^{N} \widehat{\xi_{x^2 i}^*}\pmb{u}_i\pmb{u}_i^\intercal$  with eigenvalues  $\left(\widehat{\xi_{x^2 i}^*}\right)_{1 \leq i \leq N}$ :
$$
\widehat {\xi_ {x ^ {2} i} ^ {*}} = \frac {1}{\kappa} \frac {1}{\pi \bar {\mu} _ {S} (\gamma_ {i})} \operatorname {I m} \lim  _ {z \rightarrow \gamma_ {i} - \mathrm {i} 0 ^ {+}} \frac {z - \zeta_ {1}}{\zeta_ {3}} \mathcal {G} _ {\bar {\mu} _ {S}} (z) - \frac {1}{\zeta_ {3}} \tag {10}
$$
with  $\zeta_1, \zeta_3$  as in (8), (9). Note that,  $\zeta_1, \zeta_3$  can be evaluated using the observation matrix and the knowledge of  $\mu_Y, \mu_W$ , and therefore this time the statistician does not need to know the prior of  $X$ . Furthermore, assuming that  $X$  is positive semi-definite (PSD), we can construct a sub-optimal RIE for  $X$  by using  $\sqrt{\widehat{\xi_{x^2 i}^*}}$  for the eigenvalues of the estimator.
3.2.2 Case of Gaussian  $Y$ ,  $W$
If  $Y, W$  have i.i.d. Gaussian entries with variance  $1/N$ , then  $\mathcal{C}_{\mu_Y}^{(\alpha)}(z) = \mathcal{C}_{\mu_W}^{(\alpha)}(z) = z / \alpha$ . Consequently,  $\zeta_1, \zeta_3$  can easily be computed to be  $\zeta_1 = \zeta_3 = \mathcal{G}_{\bar{\mu}_S}(z) + (1 - \alpha) / (\alpha z)$ , thus the estimator (7) can be evaluated from the observation matrix. In particular, the estimator (10) simplifies to:
$$
\widehat {\xi_ {x ^ {2} i} ^ {*}} = \frac {1}{\kappa} \left[ - 1 + \frac {1}{\alpha \left(\pi^ {2} \bar {\mu} _ {S} (\gamma_ {i}) ^ {2} + \left(\pi \mathrm {H} [ \bar {\mu} _ {S} ] (\gamma_ {i}) + \frac {1 - \alpha}{\alpha \gamma_ {i}}\right) ^ {2}\right)} \right] \tag {11}
$$
3.3 Explicit RIE for  $Y$
Our explicit RIE for  $\pmb{Y}$  is constructed as  $\widehat{\Xi_{Y}^{*}}(S) = \sum_{i=1}^{N} \widehat{\xi_{y_i}^{*}}\pmb{u}_i\pmb{v}_i^{\top}$  with singular values  $\left(\widehat{\xi_{y_i}^{*}}\right)_{1 \leq i \leq N}$ :
$$
\widehat {\xi_ {y _ {i} ^ {*}}} = \frac {1}{\sqrt {\kappa}} \frac {1}{\pi \bar {\mu} _ {S} (\gamma_ {i})} \operatorname {I m} \lim  _ {z \rightarrow \gamma_ {i} - \mathrm {i} 0 ^ {+}} q _ {4} \tag {12}
$$
where  $\gamma_{i}$  is the  $i$ -th singular value of  $S$ , and  $q_{4}$  is the solution to the following system of equations:
$$
\left\{ \begin{array}{l} \beta_ {1} = \frac {\mathcal {C} _ {\mu_ {W}} ^ {(\alpha)} \left(q _ {1} q _ {2}\right)}{q _ {1}} + \frac {1}{2} \sqrt {\frac {q _ {3}}{q _ {1}}} \left(\mathcal {R} _ {\rho_ {X}} \left(q _ {4} + \sqrt {q _ {1} q _ {3}}\right) - \mathcal {R} _ {\rho_ {X}} \left(q _ {4} - \sqrt {q _ {1} q _ {3}}\right)\right) \\ \beta_ {4} = \frac {1}{2} \left(\mathcal {R} _ {\rho_ {X}} \left(q _ {4} + \sqrt {q _ {1} q _ {3}}\right) + \mathcal {R} _ {\rho_ {X}} \left(q _ {4} - \sqrt {q _ {1} q _ {3}}\right)\right) \\ q _ {1} = \mathcal {G} _ {\bar {\mu} _ {S}} (z), \quad q _ {2} = \alpha \mathcal {G} _ {\bar {\mu} _ {S}} (z) + (1 - \alpha) \frac {1}{z} \\ q _ {3} = \frac {(z - \beta_ {1}) ^ {2}}{\beta_ {4} ^ {2}} \mathcal {G} _ {\bar {\mu} _ {S}} (z) - \frac {z - \beta_ {1}}{\beta_ {4} ^ {2}}, \quad q _ {4} = \frac {z - \beta_ {1}}{\beta_ {4}} \mathcal {G} _ {\bar {\mu} _ {S}} (z) - \frac {1}{\beta_ {4}} \end{array} \right. \tag {13}
$$
Similarly to the estimator derived for  $X$ , if  $\rho_{X}$  is a symmetric measure then the optimal singular values for the estimator of  $Y$  are all zero, see remark 5.
If  $X$  is a shifted Wigner matrix, i.e.  $\mathbf{X} = \mathbf{F} + c\mathbf{I}$  with  $\mathbf{F} = \mathbf{F}^{\intercal} \in \mathbb{R}^{N \times N}$  having i.i.d. Gaussian entries with variance  $1/N$  and  $c \neq 0$  a real number, then  $\mathcal{R}_{\rho_X}(z) = z + c$ . Moreover, if  $\mathbf{W}$  is Gaussian matrix with variance  $1/N$ , then the set of equations (13) simplifies to a great extent, and we can compute  $q_4$  analytically in terms of  $\mathcal{G}_{\bar{\mu}_S}(z)$ , see appendix D.4.
3.4 Algorithmic nature of the RIEs
The explicit RIEs (7) and (12) proposed in this section, provide spectral algorithms to estimate the matrix factors from the data matrix (and the priors). An essential ingredient that must be extracted from the data matrix is  $\mathcal{G}_{\bar{\mu}_S}(z)$ . This quantity can be approximated from the observation matrix using Cauchy kernel method introduced in [58](see section 19.5.2), from which  $\bar{\mu}_S(.)$  can be approximated using (6). Therefore, given an observation matrix  $S$ , the spectral algorithm proceeds as follows:
1. Compute the SVD of  $S$
2. Approximate  $\mathcal{G}_{\bar{\mu}_S}(z)$  from the singular values of  $S$
3. Construct the RIEs for  $X, Y$  as proposed in paragraphs 3.2, 3.3.
2Like the case for  $X$  , we omit some of the parameters which do not appear in the final estimator.
Figure 1: MSE of estimating  $\mathbf{X}$ . MSE is normalized by the norm of the signal,  $\| \mathbf{X}\|_{\mathrm{F}}^{2}$ .  $\mathbf{X}$  is a Wishart matrix with aspect ratio  $^{1 / 4}$ ,  $\mathbf{X} = \mathbf{H}\mathbf{H}^{\intercal}$  with  $\mathbf{H} \in \mathbb{R}^{N\times 4N}$  having i.i.d. Gaussian entries of variance  $^{1 / N}$ . Both  $\mathbf{Y}$  and  $\mathbf{W}$  are  $N\times M$  matrices with i.i.d. Gaussian entries of variance  $^{1 / N}$ . RIE is applied to  $N = 2000$ ,  $M = 4000$ , and the results are averaged over 10 runs (error bars are invisible).
Figure 2: MSE of estimating  $\mathbf{Y}$ . MSE is normalized by the norm of the signal,  $\| \mathbf{Y} \|_{\mathrm{F}}^2$ .  $\mathbf{Y}$  has uniform spectral density,  $\mathcal{U}([1,3])$ .  $\mathbf{X}$  is a shifted Wigner matrix with  $c = 3$ , and  $\mathbf{W}$  is a  $N \times M$  matrix with i.i.d. Gaussian entries of variance  $^1 / N$ . RIE is applied to  $N = 2000$ ,  $M = 4000$ , and the results are averaged over 10 runs (error bars are invisible).
4 Numerical results
4.1 Performance of RIE for  $X$
We consider the case where  $Y, W$  both have i.i.d. Gaussian entries of variance  $1/N$ , and  $X$  is a Wishart matrix,  $X = \mathbf{H}\mathbf{H}^{\top}$  with  $\mathbf{H} \in \mathbb{R}^{N \times 4N}$  having i.i.d. Gaussian entries of variance  $1/N$ . For various SNRs, we examine the performance of two proposed estimators, the RIE (7), and the square-root of the estimator (10) (since  $X$  is PSD), which is sub-optimal. In figure 1, the MSEs of these algorithmic estimators are compared with the one of Oracle estimator (3). We see that the average performance of the algorithmic RIE  $\widehat{\Xi_X^*}(S)$  is very close to the (optimal) Oracle estimator  $\Xi_X^*(S)$  (relative errors are small and provided in the appendices) and we believe that the slight mismatch is due to the numerical approximations and finite-size effects. Note that, although the estimator  $\sqrt{\widehat{\Xi_{X^2}^*}(S)}$  is sub-optimal, it does not use any prior knowledge of  $X$ . For more examples, details of the numerical experiments and the relative error of the estimators, we refer to appendix C.3.
4.2 Performance of RIE for  $Y$
We consider the case where  $\mathbf{W}$  has i.i.d. Gaussian entries of variance  $1 / N$ , and  $\mathbf{X}$  is a shifted Wigner matrix with  $c = 3$ . Matrix  $\mathbf{Y}$  is constructed as  $\mathbf{Y} = \mathbf{U}_Y\Sigma \mathbf{V}_Y^\intercal$  with  $\mathbf{U}_Y\in \mathbb{R}^{N\times N}$ ,  $\mathbf{V}_Y\in \mathbb{R}^{M\times M}$  are Haar distributed, and the singular values are generated independently from the uniform distribution on [1, 3]. MSEs of the RIE (12) and the Oracle estimator (5) are illustrated in figure 2. We see that the performance of the algorithmic RIE  $\widehat{\Xi_Y^*}(S)$  is very close to the optimal estimator  $\Xi_Y^*(S)$ .
Non-rotational invariant prior In another example, which we omit here, with the same settings for  $X$ ,  $W$ , we consider the case where  $Y$  is a sparse matrix with entries distributed according to Bernoulli-Rademacher prior. The RIE is not optimal in this setting (since the prior is not bi-rotational invariant), however applying a simple thresholding function on the matrix constructed by RIE yields an estimate with lower MSE. This observation suggests that for the case of general priors, the RIEs can provide a spectral initialization for more efficient estimators. For more details and examples, see appendix D.4.
4.3 Comparing RIEs of matrix factorization and denoising
The proposed RIEs, namely (7) and (12), simplify greatly when the matrices  $W$ ,  $Y$  are Gaussian, and  $X$  is a shifted Wigner matrix. We perform experiments with these priors, where for a given observation matrix  $S$ , we look at the RIEs of  $X$ ,  $Y$  for the MF problem, and simultaneously at the RIE of the product  $XY$  as a whole for the denoising problem with formulas introduced in [19] (which can also be obtained by taking  $X$  to be the identity matrix, see appendix D.3.1). Figure 3
(a) Estimating  $X$
(b) Estimating  $\mathbf{Y}$
Figure 3: MSE of factorization problem. MSE is normalized by the norm of the signal.  $\mathbf{X}$  is a shifted Wigner matrix with  $c = 1$ , and both  $\mathbf{Y}$  and  $\mathbf{W}$  are  $N \times M$  matrices with i.i.d. Gaussian entries of variance  $1/N$ . RIE is applied to  $N = 2000$ ,  $M = 4000$ . In each run, the observation matrix  $S$  is generated according to (1), and the factors  $\mathbf{X}, \mathbf{Y}$  are estimated simultaneously from  $S$ . Results are averaged over 10 runs (error bars are invisible).
(c) Estimating  $XY$
illustrates these experiments. In particular the MSE of the denoising-RIE matches well the one of the associated Oracle estimator, and as expected is lower than the MSE of the product of MF-RIEs.
5 Derivation of the explicit RIEs
In this section, we sketch the derivation of our explicit RIE for  $X$ . The RIE for  $\mathbf{Y}$  is derived similarly, but requires more involved analysis and is presented in appendix D. For simplicity, we take the SNR parameter in (1) to be 1, so the model is  $\pmb {S} = \pmb {X}\pmb {Y} + \pmb{W}$ . The optimal eigenvalues are constructed as  $\xi_{x_i}^* = \sum_{j = 1}^{N}\lambda_j\left(\pmb {u}_i^\top \pmb {x}_j\right)^2$ . We assume that in the large  $N$  limit,  $\xi_{x_i}^*$  can be approximated by its expectation and we introduce
$$
\widehat {\xi} _ {x i} ^ {*} = \sum_ {j = 1} ^ {N} \lambda_ {j} \mathbb {E} \left[ \left(\boldsymbol {u} _ {i} ^ {\mathrm {T}} \boldsymbol {x} _ {j}\right) ^ {2} \right] \tag {14}
$$
where the expectation is over the (left) singular vectors of the observation matrix  $S$ . Therefore, to compute these eigenvalues, we need to find the mean squared overlap  $\mathbb{E}\left[\left(\boldsymbol{u}_i^\top \boldsymbol{x}_j\right)^2\right]$  between eigenvectors of  $X$  and singular vectors of  $S$ . In what follows, we will see that (a rescaling of) this quantity can be expressed in terms of  $i$ -th singular value of  $S$  and  $j$ -th eigenvector of  $X$  (and the limiting measures, indeed). Thus, we will use the notation  $O_X(\gamma_i, \lambda_j) \coloneqq N\mathbb{E}\left[\left(\boldsymbol{u}_i^\top \boldsymbol{x}_j\right)^2\right]$  in the following. In the next section, we discuss how the overlap can be computed from the resolvent of the "Hermitized" version of  $S$ .
5.1 Relation between overlap and resolvent
Construct the matrix  $\mathcal{S} \in \mathbb{R}^{(N + M) \times (N + M)}$  from the observation matrix:
$$
\boldsymbol {S} = \left[ \begin{array}{c c} \mathbf {0} _ {N \times N} & \boldsymbol {S} \\ \boldsymbol {S} ^ {\intercal} & \mathbf {0} _ {M \times M} \end{array} \right]
$$
By Theorem 7.3.3 in [59],  $\pmb{S}$  has the following eigen-decomposition:
$$
\boldsymbol {S} = \left[\begin{array}{c c c}\hat {\boldsymbol {U}} _ {S}&\hat {\boldsymbol {U}} _ {S}&\mathbf {0}\\\hat {\boldsymbol {V}} _ {S} ^ {(1)}&- \hat {\boldsymbol {V}} _ {S} ^ {(1)}&\boldsymbol {V} _ {S} ^ {(2)}\end{array}\right]\left[\begin{array}{c c c}\boldsymbol {\Gamma} _ {N}&\mathbf {0}&\mathbf {0}\\\mathbf {0}&- \boldsymbol {\Gamma} _ {N}&\mathbf {0}\\\mathbf {0}&\mathbf {0}&\mathbf {0}\end{array}\right]\left[ \right.\begin{array}{c c c}\hat {\boldsymbol {U}} _ {S}&\hat {\boldsymbol {U}} _ {S}&\mathbf {0}\\\hat {\boldsymbol {V}} _ {S} ^ {(1)}&- \hat {\boldsymbol {V}} _ {S} ^ {(1)}&\boldsymbol {V} _ {S} ^ {(2)}\end{array}] ^ {\intercal} \tag {15}
$$
with  $\mathbf{V}_S = \left[ \begin{array}{cc} \mathbf{V}_S^{(1)} & \mathbf{V}_S^{(2)} \end{array} \right]$  in which  $\mathbf{V}_S^{(1)} \in \mathbb{R}^{M \times N}$ . And,  $\hat{\mathbf{V}}_S^{(1)} = \frac{1}{\sqrt{2}} \mathbf{V}_S^{(1)}$ ,  $\hat{\mathbf{U}}_S = \frac{1}{\sqrt{2}} \mathbf{U}_S$ . Eigenvalues of  $S$  are signed singular values of  $S$ , therefore the limiting eigenvalue distribution of  $S$  (ignoring zero eigenvalues) is the same as the limiting symmetrized singular value distribution of  $S$ . Define the resolvent of  $S$ ,
$$
\boldsymbol {G} _ {\mathcal {S}} (z) = \left(z \boldsymbol {I} - \boldsymbol {S}\right) ^ {- 1}
$$
We assume that as  $N \to \infty$  and  $z$  is not too close to the real axis, the matrix  $G_{\mathcal{S}}(z)$  concentrates around its mean. Consequently, the value of  $G_{\mathcal{S}}(z)$  becomes uncorrelated with the particular
realization of  $S$ . Specifically, as  $N \to \infty$ ,  $G_{S}(z)$  converges to a deterministic matrix for any fixed value of  $z \in \mathbb{C} \backslash \mathbb{R}$  (independent of  $\mathbf{N}$ ). Denote the eigenvectors of  $S$  by  $\mathbf{s}_i \in \mathbb{R}^{M + N}$ ,  $i = 1, \ldots, M + N$ . For  $z = x - \mathrm{i}\epsilon$  with  $x \in \mathbb{R}$  and small  $\epsilon$ , we have:
$$
\boldsymbol {G} _ {\mathcal {S}} (x - \mathrm {i} \epsilon) = \sum_ {k = 1} ^ {2 N} \frac {x + \mathrm {i} \epsilon}{(x - \tilde {\gamma} _ {k}) ^ {2} + \epsilon^ {2}} \mathbf {s} _ {k} \mathbf {s} _ {k} ^ {\intercal} + \frac {x + \mathrm {i} \epsilon}{x ^ {2} + \epsilon^ {2}} \sum_ {k = 2 N + 1} ^ {N + M} \mathbf {s} _ {k} \mathbf {s} _ {k} ^ {\intercal}
$$
where  $\tilde{\gamma}_k$  are the eigenvalues of  $S$ , which are in fact the (signed) singular values of  $S$ ,  $\tilde{\gamma}_1 = \gamma_1, \ldots, \tilde{\gamma}_N = \gamma_N$ ,  $\tilde{\gamma}_{N+1} = -\gamma_1, \ldots, \tilde{\gamma}_{2N} = -\gamma_N$ .
Define the vectors  $\tilde{\pmb{x}}_i = [x_i^\top, \mathbf{0}_M]^\top$  for  $\pmb{x}_i$  eigenvectors of  $\pmb{X}$ . We have
$$
\tilde {\boldsymbol {x}} _ {i} ^ {\mathsf {T}} \left(\operatorname {I m} \boldsymbol {G} _ {\mathcal {S}} (x - \mathrm {i} \epsilon)\right) \tilde {\boldsymbol {x}} _ {i} = \sum_ {k = 1} ^ {2 N} \frac {\epsilon}{(x - \tilde {\gamma} _ {k}) ^ {2} + \epsilon^ {2}} \left(\tilde {\boldsymbol {x}} _ {i} ^ {\mathsf {T}} \mathbf {s} _ {k}\right) ^ {2} + \frac {\epsilon}{x ^ {2} + \epsilon^ {2}} \sum_ {k = 2 N + 1} ^ {N + M} \left(\tilde {\boldsymbol {x}} _ {i} ^ {\mathsf {T}} \mathbf {s} _ {k}\right) ^ {2} \tag {16}
$$
Given the structure of  $\mathbf{s}_k$ 's in (15),  $\left(\tilde{\boldsymbol{x}}_i^\intercal \mathbf{s}_j\right)^2 = \frac{1}{2}\left(\boldsymbol{x}_i^\intercal \boldsymbol{u}_j\right)^2 = \left(\tilde{\boldsymbol{x}}_i^\intercal \mathbf{s}_{j + N}\right)^2$  for  $1 \leq j \leq N$ , and the second sum in (16) is zero. We assume that in the limit of large  $N$  this quantity concentrates on  $O_X(\gamma_j, \lambda_i)$  and depends only on the singular values and eigenvalue pairs  $(\gamma_j, \lambda_i)$ . We thus have:
$$
\tilde {\boldsymbol {x}} _ {i} ^ {\intercal} \left(\operatorname {I m} \boldsymbol {G} _ {\mathcal {S}} (x - \mathrm {i} \epsilon)\right) \tilde {\boldsymbol {x}} _ {i} \xrightarrow {N \rightarrow \infty} \int_ {\mathbb {R}} \frac {\epsilon}{(x - t) ^ {2} + \epsilon^ {2}} O _ {X} (t, \lambda_ {i}) \bar {\mu} _ {S} (t) d t \tag {17}
$$
where the overlap function  $O_X(t, \lambda_i)$  is extended (continuously) to arbitrary values within the support of  $\bar{\mu}_S$  (the symmetrized limiting singular value distribution of  $S$ ) with the property that  $O_X(t, \lambda_i) = O_X(-t, \lambda_i)$  for  $t \in \mathrm{supp}(\mu_S)$ . Sending  $\epsilon \to 0$ , we find
$$
\tilde {\boldsymbol {x}} _ {i} ^ {\mathsf {T}} \left(\operatorname {I m} \boldsymbol {G} _ {S} (x - \mathrm {i} \epsilon)\right) \tilde {\boldsymbol {x}} _ {i} \rightarrow \pi \bar {\mu} _ {S} (x) O _ {X} (x, \lambda_ {i}) \tag {18}
$$
This is a crucial relation as it allows us to study the overlap by means of the resolvent of  $\mathcal{S}$ . In the next section, we establish a connection between this resolvent and the signal  $X$ , which enables us to determine the optimal eigenvalues values  $\widehat{\xi}_{x_i}^*$  in terms of the singular values of  $S$ .
5.2 Resolvent relation
To derive the resolvent relation between  $S$  and  $X$ , we fix the matrix  $X$  and consider the model
$$
\boldsymbol {S} = \boldsymbol {X} \boldsymbol {U} _ {1} \boldsymbol {Y} \boldsymbol {V} _ {1} ^ {\intercal} + \boldsymbol {U} _ {2} \boldsymbol {W} \boldsymbol {V} _ {2} ^ {\intercal}
$$
with  $\mathbf{Y},\mathbf{W}\in \mathbb{R}^{N\times M}$  fixed matrices with limiting singular value distribution  $\mu_Y,\mu_W$  , and  $U_{1},U_{2}\in$ $\mathbb{R}^{N\times N}$ $V_{1},V_{2}\in \mathbb{R}^{M\times M}$  independent random Haar matrices. Indeed, if we substitute the SVD of the matrices  $\pmb{Y}$ $W$  in model (1) we find the latter model. Now, the average over the singular vectors of  $\pmb{S}$  (with fixed  $\pmb{X}$ ) is equivalent to the average over the matrices  $U_{1},U_{2},V_{1},V_{2}$ . In appendix C.1, using the Replica trick, we derive the following relation in the limit  $N\to \infty$  ..
$$
\left\langle \boldsymbol {G} _ {\mathcal {S}} (z) \right\rangle = \left[ \begin{array}{c c} \zeta_ {3} ^ {- 1} \boldsymbol {G} _ {X ^ {2}} \left(\frac {z - \zeta_ {1}}{\zeta_ {3}}\right) & \boldsymbol {0} \\ \boldsymbol {0} & (z - \zeta_ {2}) ^ {- 1} \boldsymbol {I} _ {M} \end{array} \right] \tag {19}
$$
with  $\zeta_1, \zeta_2, \zeta_3$  satisfying set of equations (41).  $\langle .\rangle$  is the expectation w.r.t. the singular vectors of  $S$  (or equivalently over  $U_1, U_2, V_1, V_2$ ), and  $G_{X^2}$  is the resolvent of  $X^2$ . As stated earlier, we assume that the resolvent  $G_S(z)$  concentrates in the limit  $N \to \infty$ , therefore we drop the brackets in the following computation.
5.3 Overlaps and optimal eigenvalues
From (18), (19), we find:
$$
\begin{array}{l} O _ {X} (\gamma , \lambda_ {i}) \approx \frac {1}{\pi \bar {\mu} _ {S} (\gamma)} \operatorname {I m} \lim  _ {z \rightarrow \gamma - \mathrm {i} 0 ^ {+}} \boldsymbol {x} _ {i} ^ {\intercal} \zeta_ {3} ^ {- 1} \boldsymbol {G} _ {X ^ {2}} \left(\frac {z - \zeta_ {1}}{\zeta_ {3}}\right) \boldsymbol {x} _ {i} \\ = \frac {1}{\pi \bar {\mu} _ {S} (\gamma)} \operatorname {I m} \lim  _ {z \rightarrow \gamma - \mathrm {i} 0 ^ {+}} \frac {1}{z - \zeta_ {1} - \zeta_ {3} \lambda_ {i} ^ {2}} \tag {20} \\ \end{array}
$$
In Fig. 4 we illustrate that the theoretical predictions (20) are in good agreement with numerical simulations for a particular case of  $\mathbf{X}$  a Wigner matrix, and  $\mathbf{Y}$ ,  $\mathbf{W}$  with i.i.d. Gaussian entries.
Figure 4: Comparison of the theoretical prediction (20) of the rescaled overlap with the numerical simulation. The rescaled overlap between 200-th and 800-th left singular vector of  $\pmb{S}$  and the eigenvectors of  $\pmb{X}$  is illustrated.  $\pmb{X} = \pmb{X}^{\intercal} \in \mathbb{R}^{N \times N}$  has i.i.d. Gaussian entries with variance  $1 / \sqrt{N}$  and is fixed. Both  $\pmb{Y}$  and  $\pmb{Z}$  are  $N \times M$  matrices with i.i.d. Gaussian entries of variance  $1 / N$ . The simulation results are average of 1000 experiments with fixed  $\pmb{X}$ , and  $N = 1000$ ,  $M = 2000$ . Some of the simulation points are dropped for clarity. One can see that the overlap is an even function of eigenvalues  $\lambda_{i}$ , so the optimal eigenvalues  $\xi_{x_i}^* = \sum_{j=1}^{N} \lambda_j (\pmb{u}_i^T \pmb{x}_j)^2$  are all zero, as discussed in remark 2.
Once we have the overlap, we can compute the optimal eigenvalues to be
$$
\widehat {\xi} _ {x i} ^ {*} \approx \frac {1}{N} \sum_ {j = 1} ^ {N} \lambda_ {j} O _ {X} \left(\gamma_ {i}, \lambda_ {j}\right) \approx \frac {1}{\pi \bar {\mu} _ {S} (\gamma_ {i})} \operatorname {I m} \lim  _ {z \rightarrow \gamma_ {i} - \mathrm {i} 0 ^ {+}} \frac {1}{N} \sum_ {j = 1} ^ {N} \frac {\lambda_ {j}}{z - \zeta_ {1} - \zeta_ {3} \lambda_ {j} ^ {2}} \tag {21}
$$
With a bit of algebra, we find the estimator in (7) in the limit  $N\to \infty$  , see appendix C.2.
We studied the MF problem with extensive-rank hidden matrices, and proposed explicit (optimal) RIEs to recover the distinct factors. The model we considered, although limited, is the first analytically solvable model in this challenging regime, and we believe it paves the way for future investigations. Extending the methodology developed here to the general case where both matrices can be nonsymmetric is an interesting research direction (note that in this case factors can be recovered up to rotations). Moreover, adapting RIEs to incorporate additional structures/ constraints on the signals is a problem with practical importance, that we leave for future investigations.
In general, the MF problem in the extensive-rank regime has not received an in-depth exploration compared to its counterpart in the low-rank regime. While substantial progress has been made in understanding and devising algorithms for low-rank matrices, the challenges posed by extensive-rank matrices remain relatively uncharted. Specifically, the study of general non-rotation invariant priors with i.i.d entries (e.g. a prior supported on the positive real line as in non-negative MF) stands out as an underdeveloped area from the theory side.
We are thankful to Jean Barbier for interesting discussions on related problems. The work of F. P has been supported by Swiss National Science Foundation grant no 200021-204119.
[1] Ivana Tošić and Pascal Frossard. Dictionary learning. IEEE Signal Processing Magazine, 28(2):27-38, 2011.
[2] Julien Mairal, Jean Ponce, Guillermo Sapiro, Andrew Zisserman, and Francis Bach. Supervised dictionary learning. Advances in neural information processing systems, 21, 2008.
[3] Bruno A Olshausen and David J Field. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature, 381(6583):607-609, 1996.
[4] Bruno A Olshausen and David J Field. Sparse coding with an overcomplete basis set: A strategy employed by v1? Vision research, 37(23):3311-3325, 1997.
[5] Kenneth Kreutz-Delgado, Joseph F Murray, Bhaskar D Rao, Kjersti Engan, Te-Won Lee, and Terrence J Sejnowski. Dictionary learning algorithms for sparse representation. Neural computation, 15(2):349-396, 2003.