SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Throughout the numerical experiments, we consider the matrix  $\mathbf{W}$  to have i.i.d. Gaussian entries with variance  $1/N$ , so  $\mathcal{C}_{\mu_W}^{(\alpha)}(z) = \frac{1}{\alpha} z$ . And,  $\mathbf{X} = \mathbf{F} + c\mathbf{I}$  where  $\mathbf{F} = \mathbf{F}^{\intercal} \in \mathbb{R}^{N \times N}$  has i.i.d. entries with variance  $1/N$ , and  $c \neq 0$  is a real number, so  $\mathcal{R}_{\rho_X}(z) = z + c$ . With these choices, the solution (76) simplifies to:
$$
\left\{ \begin{array}{l} \beta_ {1} ^ {*} = \frac {1}{\alpha} q _ {2} ^ {*} + q _ {3} ^ {*}, \quad \beta_ {2} ^ {*} = q _ {1} ^ {*}, \quad \beta_ {3} ^ {*} = q _ {1} ^ {*}, \quad \beta_ {4} ^ {*} = q _ {4} ^ {*} + c \\ q _ {1} ^ {*} = \mathcal {G} _ {\bar {\mu} S} (z), \quad q _ {2} ^ {*} = \alpha \mathcal {G} _ {\bar {\mu} S} (z) + (1 - \alpha) \frac {1}{z} \\ q _ {3} ^ {*} = \frac {(z - \beta_ {1} ^ {*}) ^ {2}}{\beta_ {4} ^ {* 2}} \mathcal {G} _ {\bar {\mu} S} (z) - \frac {z - \beta_ {1} ^ {*}}{\beta_ {4} ^ {* 2}}, \quad q _ {4} ^ {*} = \frac {z - \beta_ {1} ^ {*}}{\beta_ {4} ^ {*}} \mathcal {G} _ {\bar {\mu} S} (z) - \frac {1}{\beta_ {4} ^ {*}} \end{array} \right. \tag {81}
$$
Note that in (81),  $q_1^*, q_2^*$  are given in terms of the observation, so to find the solution we only need to find the parameters  $q_3^*, q_4^*$ . In (81), one can see that we have the relation  $q_3^* = \frac{z - \beta_1^*}{\beta_4^*} q_4^*$ . Writing the parameters  $\beta_1^*, \beta_4^*$  in terms of  $q_2^*, q_3^*, q_4^*$ , after a bit of algebra we have the following relation:
$$
q _ {3} ^ {*} = \frac {z - \frac {1}{\alpha} q _ {2} ^ {*}}{2 q _ {4} ^ {*} + c} q _ {4} ^ {*} \tag {82}
$$
In the expression for  $q_4^*$  in (81), using (82) we can rewrite  $\beta_1^*, \beta_4^*$  in terms of  $q_2^*, q_4^*$ . After some manipulations we find that  $q_4^*$  is the solution to the following cubic equation:
$$
2 x ^ {3} + 3 c x ^ {2} + \left[ c ^ {2} + 2 - \left(z - \mathcal {G} _ {\bar {\mu} _ {S}} (z) - \frac {1 - \alpha}{\alpha} \frac {1}{z}\right) \mathcal {G} _ {\bar {\mu} _ {S}} (z) \right] x - c \left[ \left(z - \mathcal {G} _ {\bar {\mu} _ {S}} (z) - \frac {1 - \alpha}{\alpha} \frac {1}{z}\right) \mathcal {G} _ {\bar {\mu} _ {S}} (z) - 1 \right] = 0 \tag {83}
$$
Based on our numerical simulations, we pick the following root for  $q_4^*$ :
$$
q _ {4} ^ {*} = - \frac {c}{2} - \frac {1 2 - 3 c ^ {2} + 6 A}{3 \sqrt [ 3 ]{B}} + \frac {\sqrt [ 3 ]{B}}{1 2} \tag {84}
$$
$$
\begin{array}{l} A = \mathcal {G} _ {\bar {\mu} _ {S}} (z) ^ {2} - \frac {\mathcal {G} _ {\bar {\mu} _ {S}} (z)}{z} \left(1 - \frac {1}{\alpha}\right) - \mathcal {G} _ {\bar {\mu} _ {S}} (z) z \\ B = - 2 1 6 c A + 4 \sqrt {4 \left(1 2 - 3 c ^ {2} + 6 A\right) ^ {3} + 5 4 ^ {2} c ^ {2} A ^ {2}} \\ \end{array}
$$
Once we have  $q_4^*$ , we can find  $q_3^*$  using (82). In the end,  $\beta_1^*, \dots, \beta_4^*$  can be evaluated. Note that, for the RIE, only  $q_4^*$  is required. Other parameters are used to evaluate the resolvent relation (72) and the overlap (77).
D.4.1 Resolvent relation
We take  $\kappa = 1$ . In model (60), without loss of generality we can consider  $\mathbf{Y}$  to be diagonal.
In figure 12,  $\mathbf{Y}$  is the diagonal matrix obtained from the singular values of a Gaussian matrix with i.i.d. entries of variance  $1 / N$ . In figure 13, the non-zero entries (on main diagonal) of  $\mathbf{Y}$  are uniformly distributed in [1, 3]. As in previous cases,  $\mu_{S}, \mathcal{G}_{\bar{\mu}_{S}}(z)$  are estimated numerically using Cauchy kernel, from which the parameters  $\beta_1^*, \dots, \beta_4^*$  are computed.
D.4.2 Overlap
To illustrate the formula for the overlap (77), we fix the matrix  $\mathbf{Y}$  and run experiments over various realization of the model (60). For each experiment, we record the overlap of  $k$ -th singular vectors left and right) of  $S$  and singular vectors of  $\mathbf{Y}$ . To compute the theoretical prediction, we evaluate the parameters  $\beta_{1}^{*}, \beta_{2}^{*}, \beta_{3}^{*}, \beta_{4}^{*}$ , for  $z = \bar{\gamma}_{k} - \mathrm{i}0^{+}$  where  $\bar{\gamma}_{k}$  is the average of  $k$ -th singular value of  $S$  in the experiments.
In figure 14a, the overlap is shown for  $\mathbf{Y}$  with i.i.d. Gaussian entries of variance  $\frac{1}{N}$ , so  $\mu_{Y}$  is the Marchenko-Pastur law with aspect-ratio  $\alpha$ . In figure 14b, matrix  $\mathbf{Y}$  is constructed as  $\mathbf{Y} = \mathbf{U}_Y \boldsymbol{\Sigma} \mathbf{V}_Y^\top$ , where  $\mathbf{U}_Y \in \mathbb{R}^{N \times N}$ ,  $\mathbf{V}_Y \in \mathbb{R}^{M \times M}$  are Haar distributed orthogonal matrices, and singular values  $\sigma_1, \dots, \sigma_N$  are chosen independently uniformly from [1, 3], so  $\mu_{Y} = \mathcal{U}([1, 3])$ .
D.4.3 RIE performance
In this section, we investigate the performance of our proposed estimators for  $\mathbf{Y}$ . To construct the RIE for  $\mathbf{Y}$ , we only need  $q_4^*$  which we use (84). We compare performances of the optimal RIE (78) with the one of oracle estimator (5).
In figures 15,16, the MSE of RIE and the oracle estimator is plotted for three cases of priors:  $\mathbf{Y}$  with Gaussian entries,  $\mathbf{Y}$  with uniform spectral density, and  $\mathbf{Y}$  with Bernoulli spectral density. In all cases, observe that the RIE has the same performance as the oracle estimator.
Effect of aspect-ratio  $\alpha$ . In Figure 17, we take  $\mathbf{Y}$  to have Gaussian entries (with variance  $\frac{1}{N}$ ), and the MSE is depicted for various values of the aspect-ratio  $\alpha$ . We see that as  $M$  increases ( $\alpha$  decreases) the estimation error (of  $\mathbf{Y}$ ) decreases.
Sparse  $Y$ : a non-rotation invariant example. We consider  $Y$  to have i.i.d. entries from the Bernoulli-Rademacher distribution,
$$
Y _ {i, j} = \left\{ \begin{array}{l l} + \frac {1}{\sqrt {N}} & \text {w i t h p r o b a b i l i t y} \frac {1 - p}{2} \\ 0 & \text {w i t h p r o b a b i l i t y} p \\ - \frac {1}{\sqrt {N}} & \text {w i t h p r o b a b i l i t y} \frac {1 - p}{2} \end{array} , \quad \forall \quad 1 \leq i \leq N, \quad 1 \leq j \leq M \right.
$$
With the normalization  $1 / \sqrt{N}$ , the spectrum of  $\mathbf{Y}$  does not grow with the dimension and has a finite support, thus we can apply our estimator to reconstruct  $\mathbf{Y}$ . Note that the prior of  $\mathbf{Y}$  is not rotationally invariant, and neither the oracle estimator nor the RIE are optimal. Therefore, taking the prior into account, we apply a thresholding function on the entries of the matrix obtained from the RIE,  $\widehat{\Xi_Y^*} (S)$ . We apply the following function on each entry of the estimator:
$$
f _ {h} (x) = \left\{ \begin{array}{l l} + \frac {1}{\sqrt {N}} & \text {i f} x > \frac {h}{\sqrt {N}} \\ 0 & \text {i f} | x | \leq \frac {h}{\sqrt {N}} \\ - \frac {1}{\sqrt {N}} & \text {i f} x <   - \frac {h}{\sqrt {N}} \end{array} \right., \qquad \text {f o r} h \in [ 0, 1 ]
$$
In figure 18, the MSE of the oracle estimator, RIE, and  $\mathrm{RIE} + f_p(x)$  (with  $h = p$ ) is plotted. A few remarks on this figure are in order. First, RIEs are not limited to rotationally invariant priors and can give non-trivial estimates for non-rotationally invariant priors, although they are sub-optimal. The
(b) Entry  $i = 1, j = 2$ , non-diagonal entry in the upper-left block
(c) Entry  $i = j = 2001$ , first diagonal entry in the lower-right block
(d) Entry  $i = 2001, j = 2002$ , non-diagonal entry in the lower-right block
(e) Entry  $i = 1, j = 2001$ , first entry in the upper-right block
(f) Entry  $i = 1, j = 2002$ , second entry in the upper-right block
Figure 12: Illustration of (72).  $\mathbf{Y} \in \mathbb{R}^{N \times M}$  is a diagonal matrix obtained from the singular values of a  $N \times M$  matrix with i.i.d. entries of variance  $1/N$ ,  $\mathbf{X} = \mathbf{X}^{\top}$  is shifted Wigner matrix with  $c = 3$ , and  $\mathbf{Z}$  is a Gaussian matrices with. The empirical estimate of  $\mathbf{G}_S(z)$  (dashed blue line) is computed for  $z = \gamma_i - \mathrm{i}\sqrt{\frac{1}{2N}}$  for  $1 \leq i \leq N$ , for  $N = 2000, M = 4000$ . Theoretical one (solid orange line) is computed from the rhs of (72) with parameters computed from the generated matrix. Note that, the theoretical one has also fluctuations because the parameters  $\beta_1^*, \dots, \beta_4^*$  are computed from the numerical estimate of  $\mathcal{G}_{\bar{\mu}_S}(z)$ .
(d) Entry  $i = 2001$ ,  $j = 2002$ , non-diagonal entry in the lower-right block
(e) Entry  $i = 1, j = 2001$ , first entry in the upper-right block
(f) Entry  $i = 1, j = 2002$ , second entry in the upper-right block
Figure 13: Illustration of (72).  $\mathbf{Y} \in \mathbb{R}^{N \times M}$  is a diagonal matrix with (main) diagonal entries uniformly distributed in  $[1,3]$ ,  $\mathbf{X} = \mathbf{X}^{\intercal}$  is shifted Wigner matrix with  $c = 3$ , and  $\mathbf{Z}$  is a Gaussian matrices with. The empirical estimate of  $G_{S}(z)$  (dashed blue line) is computed for  $z = \gamma_{i} - \mathrm{i}\sqrt{\frac{1}{2N}}$  for  $1 \leq i \leq N$ , for  $N = 2000$ ,  $M = 4000$ . Theoretical one (solid orange line) is computed from the rhs of (72) with parameters computed from the generated matrix. Note that, the theoretical one has also fluctuations because the parameters  $\beta_{1}^{*}, \dots, \beta_{4}^{*}$  are computed from the numerical estimate of  $\mathcal{G}_{\bar{\mu}_S}(z)$ .
(a) Gaussian  $\mathbf{Y}$
(b)  $\mathbf{Y}$  with Uniform spectrum,  $\mathcal{U}\big([1,3]\big)$
Figure 14: Computation of the rescaled overlap.  $\mathbf{X}$  is a shifted Wigner matrix with  $c = 3$ , and  $\mathbf{W}$  has i.i.d. Gaussian entries of variance  $1 / N$ , and  $N / M = 1 / 2$ . The simulation results are average of 1000 experiments with fixed  $\mathbf{Y}$ , and  $N = 1000$ ,  $M = 2000$ . Some of the simulation points are dropped for clarity.
(a) Gaussian
(b) Uniform spectral,  $\mathcal{U}([1,3])$
Figure 15: Estimating  $\mathbf{Y}$ . MSE is normalized by the norm of the signal,  $\| \mathbf{Y} \|_{\mathrm{F}}^2$ .  $\mathbf{X}$  is a shifted Wigner matrix with  $c = 3$ , and  $\mathbf{W}$  has i.i.d. Gaussian entries of variance  $1/N$ , and  $N/M = 1/2$ . The RIE is applied to  $N = 2000$ ,  $M = 4000$ , and the results are averaged over 10 runs (error bars are invisible). Average relative error between  $\mathrm{RIE} \widehat{\Xi_Y^*}(S)$  and Oracle estimator is also reported.
Figure 16: Estimating  $\mathbf{Y}$  with Bernoulli spectral prior. MSE is normalized by the norm of the signal,  $\| \mathbf{Y}\|_{\mathrm{F}}^{2}$ .  $\mathbf{Y}$  has Bernoulli spectral distribution with parameter  $p$ .  $\mathbf{X}$  is a shifted Wigner matrix with  $c = 3$ , and  $\mathbf{W}$  has i.i.d. Gaussian entries of variance  $1 / N$ , and  $N / M = 1 / 2$ . The RIE is applied to  $N = 2000$ ,  $M = 4000$ , and the results are averaged over 10 runs (error bars are invisible). Average relative error between RIE  $\widehat{\Xi}_{Y}^{*}(\mathbf{S})$  and Oracle estimator is also reported.
Figure 17: MSE of estimating  $\mathbf{Y}$  as a function of aspect-ratio  $\alpha$ ,  $\mathbf{Y}$  has Gaussian entries of variance  $1/N$ , and  $\kappa = 5$ . MSE is normalized by the norm of the signal,  $\|\mathbf{Y}\|_{\mathrm{F}}^2$ .  $\mathbf{X}$  is a shifted Wigner matrix with  $c = 3$ , and  $\mathbf{W}$  has i.i.d. Gaussian entries of variance  $1/N$ . The RIE is applied to  $N = 2000$ ,  $M = 1/\alpha N$ , and the results are averaged over 10 runs (error bars are invisible). Average relative error between RIE  $\widehat{\Xi}_{Y}^{*}(\mathbf{S})$  and Oracle estimator is also reported.
(a)  $p = 0.5$
(b)  $p = 0.9$
Figure 18: Estimating  $\mathbf{Y}$  with Bernoulli-Rademacher entries. MSE is normalized by the norm of the signal,  $\| \mathbf{Y} \|_{\mathrm{F}}^{2}$ .  $\mathbf{X}$  is a shifted Wigner matrix with  $c = 3$ , and  $\mathbf{W}$  has i.i.d. Gaussian entries of variance  $1/N$ , and  $N/M = 1/2$ . The RIE is applied to  $N = 2000$ ,  $M = 4000$ , and the results are averaged over 10 runs (error bars are invisible).
RIE's output can be refined, or used as a warmed-up initialization for other algorithms to get a better estimate.
In figure 19, for one experiment, the MSE is plotted for RIE and  $\mathrm{RIE} + f(x)$  with the best  $h$  among  $\{0,0.1,\dots ,1\}$ . We observe that for the particular case of Bernoulli-Rademacher prior, the thresholding stage can improve the MSE when SNR is greater than 1, however the parameter  $h$  should be chosen properly.
E Comparison of RIEs for MF and denoising
For estimating  $X$ , we have derived the estimator (49) for general priors  $\rho_{X}, \mu_{Y}, \mu_{W}$ . This estimator simplifies greatly, with parameters in (52), when both  $\mu_{Y}, \mu_{W}$  are Marchenko-Pastur distribution, i.e. both  $Y, W$  having i.i.d. Gaussian entries of variance  $1/N$ . Similarly, although the RIE for  $Y$  in (78) is derived for the general priors, it reduces to a rather simple estimator if  $\rho_{X}, \mu_{W}$  are taken to be shifted Wigner, and Marchenko-Pastur distribution, respectively. Therefore, in our numerical examples on factorization problem, we consider  $X$  to be a shifted Wigner matrix, and  $Y, W$  to be Gaussian matrices.
In each experiment, the factors  $X$ ,  $Y$  are estimated simultaneously using RIE from the observation matrix  $S$ . In addition to the MSE of estimating each factor, we also compute the MSE of estimating the product  $XY$ . We compare the MSE of the product with the MSE of the oracle estimator and the RIE introduced in [19] for the denoising problem. The oracle estimator for the denoising is
(b)  $p = 0.9$
Figure 19: Estimating  $\mathbf{Y}$  with Bernoulli-Rademacher entries. MSE is normalized by the norm of the signal,  $\| \mathbf{Y} \|_{\mathrm{F}}^2$ .  $\mathbf{X}$  is a shifted Wigner matrix with  $c = 3$ , and  $\mathbf{W}$  has i.i.d. Gaussian entries of variance  $1/N$ , and  $N/M = 1/2$ . The RIE is applied to  $N = 2000$ ,  $M = 4000$ , and thresholding function is applied with the best  $h$  among  $\{0, 0.1, \dots, 1\}$ . Results are averaged over 10 runs (error bars are invisible).
(a) Estimating  $X$
(b) Estimating  $\mathbf{Y}$
Figure 20: MSE of factorization problem. MSE is normalized by the norm of the signal.  $\mathbf{X}$  is a shifted Wigner matrix with  $c = 1$ , and both  $\mathbf{Y}$  and  $\mathbf{W}$  are  $N \times M$  matrices with i.i.d. Gaussian entries of variance  $1 / N$ , and  $N / M = 1 / 2$ . The RIE is applied to  $N = 2000$ ,  $M = 4000$ . In each run, the observation matrix  $S$  is generated according to (1), and the factors  $\mathbf{X}, \mathbf{Y}$  are estimated simultaneously from  $S$ . Results are averaged over 10 runs (error bars are invisible). Average relative error between RIEs and Oracle estimators is also reported.
constructed as:
$$
\boldsymbol {\Xi} _ {X Y} ^ {*} (\boldsymbol {S}) = \sum_ {i = 1} ^ {N} \xi_ {x y _ {i}} ^ {*} \boldsymbol {u} _ {i} \boldsymbol {v} _ {i} ^ {\intercal}, \quad \xi_ {x y _ {i}} ^ {*} = \boldsymbol {u} _ {i} ^ {\intercal} \boldsymbol {X} \boldsymbol {Y} \boldsymbol {v} _ {i} \tag {85}
$$
where  $\pmb{u}_i, \pmb{v}_i$ 's are left/right singular vectors of  $S$ . In the RIE proposed in [19], the singular values are estimated by (see section D.3.1)
$$
\begin{array}{l} \widehat {\xi_ {x y _ {i}} ^ {*}} = \frac {1}{\sqrt {\kappa}} \bigg [ \gamma_ {i} - \frac {1}{\pi \bar {\mu} _ {S} (\gamma_ {i})} \mathrm {I m} \mathcal {C} _ {\mu_ {W}} ^ {(\alpha)} \bigg (\frac {1 - \alpha}{\gamma_ {i}} \pi \mathsf {H} [ \bar {\mu} _ {S} ] (\gamma_ {i}) + \alpha \big (\pi \mathsf {H} [ \bar {\mu} _ {S} ] (\gamma_ {i}) \big) ^ {2} - \alpha \big (\pi \bar {\mu} _ {S} (\gamma_ {i}) \big) ^ {2} \\ \left. + \mathrm {i} \pi \bar {\mu} _ {S} (\gamma_ {i}) \left(\frac {1 - \alpha}{\gamma_ {i}} + 2 \alpha \pi \mathsf {H} [ \bar {\mu} _ {S} ] (\gamma_ {i})\right)\right) \Bigg ] \tag {86} \\ \end{array}
$$
Note that, in general the MSE of the denoising RIE  $\widehat{\Xi_{XY}^*}(S)$ , is less than the MSE of the prduct of the estimated factors  $\widehat{\Xi_X^*(S)}\widehat{\Xi_Y^*(S)}$ .
In figures 20,21, the MSE of estimating the factors is illustrated for  $c = 1$  and  $c = 3$  respectively. The MSE of estimating the product is shown in figure 22.
(b) Estimating  $\mathbf{Y}$
Figure 21: MSE of factorization problem. MSE is normalized by the norm of the signal.  $\mathbf{X}$  is a shifted Wigner matrix with  $c = 3$ , and both  $\mathbf{Y}$  and  $\mathbf{W}$  are  $N \times M$  matrices with i.i.d. Gaussian entries of variance  $1/N$ , and  $N/M = 1/2$ . The RIE is applied to  $N = 2000$ ,  $M = 4000$ . In each run, the observation matrix  $S$  is generated according to (1), and the factors  $\mathbf{X}, \mathbf{Y}$  are estimated simultaneously from  $S$ . Results are averaged over 10 runs (error bars are invisible). Average relative error between RIEs and Oracle estimators is also reported.
(a)  $c = 1$
(b)  $c = 3$
Figure 22: MSE of the product of the factors. MSE is normalized by the norm of the signal  $\| X\mathbf{Y}\|_{\mathrm{F}}^2$ .  $\mathbf{X}$  is a shifted Wigner matrix with  $c = 1, c = 3$ , and both  $\mathbf{Y}$  and  $\mathbf{W}$  are  $N\times M$  matrices with i.i.d. Gaussian entries of variance  $1 / N$ , and  $N / M = 1 / 2$ . The RIE is applied to  $N = 2000$ ,  $M = 4000$ . Results are averaged over 10 runs (error bars are invisible).
F Case of  $\alpha \geq 1$
In this section we consider the case where  $M \leq N$  and  $N / M \to \alpha \geq 1$  as  $N \to \infty$ . Throughout this section  $\Gamma \in \mathbb{R}^{N \times M}$  is a (tall) matrix with  $\Gamma_M$  in its upper  $M \times M$  block, and the rest zero entries.  $\Gamma_M$  is diagonal matrix constructed from  $\gamma \in \mathbb{R}^M$  which are the singular values of  $S$ .
Similar to the case of  $\alpha \leq 1$ , resolvent of the matrix  $\pmb{S} \in \mathbb{R}^{(N + M) \times (N + M)}$  plays a central role in deriving the RIEs. For the case of  $M \geq N$ , with  $\pmb{S} = \pmb{U}_S \pmb{\Gamma} \pmb{V}_S^\top$ , the matrix  $\pmb{S}$  has the following eigen-decomposition:
$$
\boldsymbol {S} = \left[ \begin{array}{c c c} \hat {\boldsymbol {U}} _ {S} ^ {(1)} & - \hat {\boldsymbol {U}} _ {S} ^ {(1)} & \boldsymbol {U} _ {S} ^ {(2)} \\ \hat {\boldsymbol {V}} _ {S} & - \hat {\boldsymbol {V}} _ {S} & \boldsymbol {0} \end{array} \right] \left[ \begin{array}{c c c} \boldsymbol {\Gamma} _ {M} & \boldsymbol {0} & \boldsymbol {0} \\ \boldsymbol {0} & - \boldsymbol {\Gamma} _ {M} & \boldsymbol {0} \\ \boldsymbol {0} & \boldsymbol {0} & \boldsymbol {0} \end{array} \right] \left[ \begin{array}{c c c} \hat {\boldsymbol {U}} _ {S} ^ {(1)} & - \hat {\boldsymbol {U}} _ {S} ^ {(1)} & \boldsymbol {U} _ {S} ^ {(2)} \\ \hat {\boldsymbol {V}} _ {S} & - \hat {\boldsymbol {V}} _ {S} & \boldsymbol {0} \end{array}\right] ^ {\intercal} \tag {87}
$$
with  $\boldsymbol{U}_S = \left[ \begin{array}{cc}\boldsymbol{U}_S^{(1)} & \boldsymbol{U}_S^{(2)} \end{array} \right]$  in which  $\boldsymbol{U}_S^{(1)}\in \mathbb{R}^{N\times M}$ . And,  $\hat{\boldsymbol{U}}_S^{(1)} = \frac{1}{\sqrt{2}}\boldsymbol{U}_S^{(1)},\hat{\boldsymbol{V}}_S = \frac{1}{\sqrt{2}}\boldsymbol{V}_S$ . The resolvent of  $\boldsymbol{S}$  can be written as:
$$
\boldsymbol {G} _ {\mathcal {S}} (x - \mathrm {i} \epsilon) = \sum_ {k = 1} ^ {2 M} \frac {x + \mathrm {i} \epsilon}{(x - \tilde {\gamma} _ {k}) ^ {2} + \epsilon^ {2}} \mathbf {s} _ {k} \mathbf {s} _ {k} ^ {\intercal} + \frac {x + \mathrm {i} \epsilon}{x ^ {2} + \epsilon^ {2}} \sum_ {k = 2 M + 1} ^ {M + N} \mathbf {s} _ {k} \mathbf {s} _ {k} ^ {\intercal}
$$
where  $\tilde{\gamma}_k$  are the eigenvalues of  $S$ , which are in fact the (signed) singular values of  $S$ ,  $\tilde{\gamma}_1 = \gamma_1, \ldots, \tilde{\gamma}_M = \gamma_M$ ,  $\tilde{\gamma}_{M+1} = -\gamma_1, \ldots, \tilde{\gamma}_{2M} = -\gamma_M$ .
F.1 Estimating  $X$
The RIE for  $X$  is constructed in the same way as in the case of  $\alpha \leq 1$ , (2). However, in the present case the observation matrix  $S$  has  $M$  (non-trivially zero) singular values and we need to estimate  $N$  eigenvalues for the RIE. As it will be clear, the  $N - M$  eigenvalues are chosen to be equal.
F.1.1 Relation between overlap and the resolvent
Define the vectors  $\tilde{\pmb{x}}_i = [\pmb{x}_i^\top, \pmb{0}_M]^\top$  for  $\pmb{x}_i$  eigenvectors of  $\pmb{X}$ . We have
$$
\tilde {\boldsymbol {x}} _ {i} ^ {\mathsf {T}} \left(\operatorname {I m} \boldsymbol {G} _ {\mathcal {S}} (x - \mathrm {i} \epsilon)\right) \tilde {\boldsymbol {x}} _ {i} = \sum_ {k = 1} ^ {2 M} \frac {\epsilon}{(x - \tilde {\gamma} _ {k}) ^ {2} + \epsilon^ {2}} \left(\tilde {\boldsymbol {x}} _ {i} ^ {\mathsf {T}} \mathbf {s} _ {k}\right) ^ {2} + \frac {\epsilon}{x ^ {2} + \epsilon^ {2}} \sum_ {k = 2 M + 1} ^ {M + N} \left(\tilde {\boldsymbol {x}} _ {i} ^ {\mathsf {T}} \mathbf {s} _ {k}\right) ^ {2} \tag {88}
$$
Given the structure of  $\mathbf{s}_k$ 's in (87), we have:
$$
\left(\tilde {\boldsymbol {x}} _ {i} ^ {\intercal} \mathbf {s} _ {k}\right) ^ {2} = \left\{ \begin{array}{l l} \frac {1}{2} \left(\boldsymbol {x} _ {i} ^ {\intercal} \boldsymbol {u} _ {k}\right) ^ {2} & \text {f o r} 1 \leq k \leq M \\ \frac {1}{2} \left(\boldsymbol {x} _ {i} ^ {\intercal} \boldsymbol {u} _ {k - M}\right) ^ {2} & \text {f o r} M + 1 \leq k \leq 2 M \\ \left(\boldsymbol {x} _ {i} ^ {\intercal} \boldsymbol {u} _ {k - M}\right) ^ {2} & \text {f o r} 2 M + 1 \leq k \leq M + N \end{array} \right.
$$
We assume that in the limit of large  $\mathbf{N}$  this quantity concentrates on  $O_{X}(\gamma_{j},\lambda_{i})$  and depends only on the singular values and eigenvalue pairs  $(\gamma_j,\lambda_i)$ . This assumption implies that the singular vectors associated with 0 singular values  $(\pmb {u}_j$  for  $M + 1\leq j\leq N$ ) all have the same overlap with the eigenvectors of  $X$ ,  $O_{X}(0,\lambda_{i})$ . We thus have:
$$
\tilde {\boldsymbol {x}} _ {i} ^ {\intercal} \left(\operatorname {I m} \boldsymbol {G} _ {\mathcal {S}} (x - \mathrm {i} \epsilon)\right) \tilde {\boldsymbol {x}} _ {i} \xrightarrow {N \rightarrow \infty} \frac {1}{\alpha} \int_ {\mathbb {R}} \frac {\epsilon}{(x - t) ^ {2} + \epsilon^ {2}} O _ {X} (t, \lambda_ {i}) \bar {\mu} _ {S} (t) d t + \left(1 - \frac {1}{\alpha}\right) \frac {\epsilon}{x ^ {2} + \epsilon^ {2}} O _ {X} (0, \lambda_ {i}) \tag {89}
$$
$$
\tilde {\boldsymbol {x}} _ {i} ^ {\intercal} \left(\operatorname {I m} \boldsymbol {G} _ {S} (x - \mathrm {i} \epsilon)\right) \tilde {\boldsymbol {x}} _ {i} \rightarrow \frac {1}{\alpha} \pi \bar {\mu} _ {S} (x) O _ {X} (x, \lambda_ {i}) + \left(1 - \frac {1}{\alpha}\right) \pi \delta (x) O _ {X} (x, \lambda_ {i}) \tag {90}
$$
F.1.2 Resolvent relation
We derive the resolvent relation for the same model as in (29). The derivation is similar to the procedure explained in section C.1, and we omit here. The final resolvent relation is the same as (42), with parameters satisfying:
$$
\left\{ \begin{array}{l} \zeta_ {1} ^ {*} = \frac {1}{\alpha} \frac {\mathcal {C} _ {\mu_ {W}} ^ {(1 / \alpha)} \left(p _ {1} ^ {*} p _ {2} ^ {*}\right)}{p _ {1} ^ {*}}, \quad \zeta_ {2} ^ {*} = \frac {1}{p _ {2} ^ {*}} \left(\mathcal {C} _ {\mu_ {W}} ^ {(1 / \alpha)} \left(p _ {1} ^ {*} p _ {2} ^ {*}\right) + \mathcal {C} _ {\mu_ {Y}} ^ {(1 / \alpha)} \left(p _ {2} ^ {*} p _ {3} ^ {*}\right)\right), \quad \zeta_ {3} ^ {*} = \frac {1}{\alpha} \frac {\mathcal {C} _ {\mu_ {Y}} ^ {(1 / \alpha)} \left(p _ {2} ^ {*} p _ {3} ^ {*}\right)}{p _ {3} ^ {*}} \\ p _ {1} ^ {*} = \frac {1}{\zeta_ {3} ^ {*}} \mathcal {G} _ {\rho_ {X ^ {2}}} \left(\frac {z - \zeta_ {1} ^ {*}}{\zeta_ {3} ^ {*}}\right), \quad p _ {2} ^ {*} = \frac {1}{z - \zeta_ {2} ^ {*}}, \quad p _ {3} ^ {*} = \frac {z - \zeta_ {1} ^ {*}}{\zeta_ {3} ^ {* 2}} \mathcal {G} _ {\rho_ {X ^ {2}}} \left(\frac {z - \zeta_ {1} ^ {*}}{\zeta_ {3} ^ {*}}\right) - \frac {1}{\zeta_ {3} ^ {*}} \end{array} \right. \tag {91}
$$
Again, with the same procedure as (43),(44), the saddle point equations (91) can be rewritten in a simplified form, which does not involve  $\rho_{X^2}$ , as: