SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[1] Deeksha Adil, Richard Peng, and Sushant Sachdeva. Fast, provably convergent IRLS algorithm for p-norm linear regression. Advances in Neural Information Processing Systems (NeurIPS 2019), 2019.
[2] Khurrum Aftab and Richard Hartley. Convergence of iteratively re-weighted least squares to robust m-estimators. In IEEE Winter Conference on Applications of Computer Vision (WACV 2015), pages 480–487, 2015.
[3] Ali Ahmed, Benjamin Recht, and Justin Romberg. Blind deconvolution using convex programming. IEEE Transactions on Information Theory, 60(3):1711-1732, 2014.
[4] Sohail Bahmani and Justin Romberg. Near-optimal estimation of simultaneously sparse and low-rank matrices from nested linear measurements. Information and Inference: A Journal of the IMA, 5(3):331-351, 2016.
[5] Amir Beck and Shoham Sabach. Weiszfeld's method: Old and new results. Journal of Optimization Theory and Applications, 164(1):1-40, 2015.
[6] Martin Benning and Martin Burger. Modern regularization methods for inverse problems. Acta Numerica, 27:1-111, 2018.
[7] Mario Bertero, Patrizia Boccacci, and Christine De Mol. Introduction to inverse problems in imaging. CRC Press, 2021.
[8] Nicolas Boumal. An introduction to optimization on smooth manifolds. Cambridge University Press, 2023.
[9] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems (NeurIPS 2020), 33:1877-1901, 2020.
[10] Alfred M. Bruckstein, David L. Donoho, and Michael Elad. From sparse solutions of systems of equations to sparse modeling of signals and images. SIAM Review, 51(1):34-81, 2009.
[11] T Tony Cai, Xiaodong Li, and Zongming Ma. Optimal rates of convergence for noisy sparse phase retrieval via thresholded wirtinger flow. The Annals of Statistics, pages 2221-2251, 2016.
[12] T Tony Cai and Anru Zhang. Rate-optimal perturbation bounds for singular subspaces with applications to high-dimensional statistics. The Annals of Statistics, 46(1):60-89, 2018.
[13] Patrizio Campisi and Karen Egiazarian. Blind Image Deconvolution: Theory and Applications. CRC Press, 2017.
[14] E. J. Candès, Y. Eldar, T. Strohmer, and V. Voroninski. Phase Retrieval via Matrix Completion. SIAM J. Imag. Sci., 6(1):199-225, 2013.
[15] E. J. Candès and Y. Plan. Tight Oracle Inequalities for Low-Rank Matrix Recovery From a Minimal Number of Noisy Random Measurements. IEEE Trans. Inf. Theory, 57(4):2342-2359, 2011.
[16] Venkat Chandrasekaran, Benjamin Recht, Pablo A Parrilo, and Alan S Willsky. The convex geometry of linear inverse problems. Foundations of Computational Mathematics, 12(6):805-849, 2012.
Supplementary material for Recovering Simultaneously Structured Data via Non-Convex Iteratively Reweighted Least Squares
This supplement is structured as follows.
- Appendix A presents some details about the experimental setup as well as additional numerical experiments.
- Appendix B.1 presents the proof of Theorem 2.5.
- Appendix B.2 presents the proof of Theorem 2.6.
- Appendix C details some technical results that are used in Appendices B.1 and B.2.
A Experimental Setup and Supplementary Experiments
In this section, we elaborate on the detailed experimental setup that was used in Section 4 of the main paper. Furthermore, we provide additional experiments comparing the behavior of the three methods studied in Section 4 for linear measurement operators  $\mathcal{A}$  that are closer to operators that can be encountered in applications of simultaneous low-rank and group-sparse recovery. Finally, we shed light on the evolution of the objective function (4) of IRLS (Algorithm 1), including in situations where the algorithm does not manage to recover the ground truth.
A.1 Experimental Setup
The experiments of Section 4 were conducted using MATLAB implementations of the three algorithms on different Linux machines using MATLAB versions R2019b or R2022b. In total, the preparation and execution of the experiments used approximately 1200 CPU hours. The CPU models used in the simulations are Dual 18-Core Intel Xeon Gold 6154, Dual 24-Core Intel Xeon Gold 6248R, Dual 8-Core Intel Xeon E5-2667, 28-Core Intel Xeon E5-2690 v3, 64-Core Intel Xeon Phi KNL 7210-F. For Sparse Power Factorization (SPF) [60], we used our custom implementation of [60, Algorithm 4 "rSPF_HTP"] and for Riemannian adaptive iterative hard thresholding (RiemAdaIHT) [29], we used an implementation provided to us by Max Pfeffer in private communications. We refer to Section 3 for implementation details for the IRLS method Algorithm 1.
In all phase transition experiments, we define successful recovery such that the relative Frobenius error  $\frac{\|\mathbf{X}^{(K)} - \mathbf{X}_{\star}\|_F}{\|\mathbf{X}_{\star}\|_F}$  of the iterate  $\mathbf{X}^{(K)}$  returned by the algorithm relative to the simultaneously low-rank and row-sparse ground truth matrix  $\mathbf{X}_{\star}$  is smaller than the threshold  $10^{-4}$ . As stopping criteria, we used the criterion that the relative change of Frobenius norm satisfies  $\frac{\|\mathbf{X}^{(k)} - \mathbf{X}^{(k-1)}\|_F}{\|\mathbf{X}^{(k)}\|_F} < \mathrm{tol}$  for IRLS, the change in the matrix factors norms satisfy  $\|\mathbf{U}_k - \mathbf{U}_{k-1}\| < \mathrm{tol}$  and  $\|\mathbf{V}_k - \mathbf{V}_{k-1}\| < \mathrm{tol}$  for SPF, and the norm of the Riemannian gradient in RiemAdaIHT being smaller than tol for  $\mathrm{tol} = 10^{-10}$ , or if a maximal number of iterations is reached. This iteration threshold was chosen as max_iter = 250 for IRLS and SPF and as max_iter = 2000 for RiemAdaIHT, reflecting the fact that RiemAdaIHT is a gradient-type method which might need many iterations to reach a high-accuracy solution. The parameters were chosen so that the stopping criteria do not prevent a method's iterates reaching the recovery threshold if they were to reach  $\mathbf{X}_{\star}$  eventually.
In the experiments, we chose random ground truths  $\mathbf{X}_{\star} \in \mathbb{R}^{n_1 \times n_2}$  of rank  $r$  and row-sparsity  $s$  such that  $\mathbf{X}_{\star} = \tilde{\mathbf{X}}_{\star} / \left\| \tilde{\mathbf{X}}_{\star} \right\|_F$ , where  $\tilde{\mathbf{X}}_{\star} = \mathbf{U}_{\star} \mathrm{diag}(\mathbf{d}_{\star}) \mathbf{V}_{\star}^*$ , and where  $\mathbf{U}_{\star} \in \mathbb{R}^{n_1 \times r}$  is a matrix with  $s$  non-zero rows whose location is chosen uniformly at random and whose entries are drawn from i.i.d. standard Gaussian random variables,  $\mathbf{d}$  has i.i.d. standard Gaussian entries and  $\mathbf{V}_{\star} \in \mathbb{R}^{n_2 \times r}$  has likewise i.i.d. standard Gaussian entries.
A.2 Random Rank-One Measurements
In Section 4, we considered only measurement operator  $\mathcal{A}:\mathbb{R}^{n_1\times n_2}\to \mathbb{R}^m$  whose matrix representation consists of i.i.d. Gaussian entries, i.e., operators such that there are independent matrices
$\mathbf{A}_1, \ldots, \mathbf{A}_m$  with i.i.d. standard Gaussian entries such that
$$
\mathcal {A} (\mathbf {X}) _ {j} = \left\langle \mathbf {A} _ {j}, \mathbf {X} \right\rangle_ {F}
$$
for any  $\mathbf{X} \in \mathbb{R}^{n_1 \times n_2}$ . While it is known that such Gaussian measurement operators satisfy the  $(r, s)$ -RIP of Section 4, which is the basis of our convergence theorem Theorem 2.5, in a regime of a near-optimal number of measurements with high probability, practically relevant measurement operators are often more structured; another downside of dense Gaussian measurements is that it is computationally expensive to implement their action on matrices.
In relevant applications of our setup, however, e.g., in sparse phase retrieval [46, 11, 47] or blind deconvolution [59, 83], the measurement operator consists of rank-one measurements. For this reason, we now conduct experiments in settings related to the ones depicted in Figure 1 and Figure 2 Section 4, but for random rank-one measurements where the action of  $\mathcal{A}:\mathbb{R}^{n_1\times n_2}\to \mathbb{R}^m$  on  $\mathbf{X}$  can be written as
$$
\mathcal {A} (\mathbf {X}) _ {j} = \left\langle \mathbf {a} _ {j} \mathbf {b} _ {j} ^ {*}, \mathbf {X} \right\rangle_ {F}
$$
for each  $j = 1, \dots, m$ , where  $\mathbf{a}_j, \mathbf{b}_j$  are independent random standard Gaussian vectors. In Figure 4, we report the phase transition performance of RiemAdaIHT, SPF and IRLS for  $(256 \times 40)$ -dimensional ground truths of different row-sparsities and different ranks if we are given such random rank-one measurements.
Figure 4: Left column: RiemAdaiHT, center: SPF, right: IRLS. Success rates for the recovery of low-rank and row-sparse matrices from random rank-one measurements. First row: Rank-1 ground truth  $\mathbf{X}_{\star}$  (cf. Figure 1. Second row: Rank-5 ground truth  $\mathbf{X}_{\star}$  (cf. Figure 2).
We observe in Figure 4 that compared to the setting of dense Gaussian measurements, the phase transitions of all three algorithms deteriorate slightly; especially for  $r = 1$ , one can observe that the transition between no success and high empirical success rate is extends across a larger area. IRLS performs clearly best for both  $r = 1$  and  $r = 5$ , whereas SPF has the second best performance for  $r = 5$ . For  $r = 1$ , it is somewhat unclear whether RiemAdaIHT or SPF performs better.
A.3 Discrete Fourier Rank-One Measurements
We now revisit the experiments of Appendix A.2 for a third measurement setup motivated from blind deconvolution problems [3, 59, 64, 66, 83, 29], which are prevalent in astronomy, medical imaging and communications engineering [49, 13]. In particular, in these settings, if  $\mathbf{z} \in \mathbb{R}^m$  is an (unknown) signal and  $\mathbf{w} \in \mathbb{R}^m$  is an (unknown) convolution kernel, assume we are given the entries of their convolution  $\widetilde{y} = \mathbf{z} * \mathbf{w}$ . If we know that  $\mathbf{z} = \mathbf{A}\mathbf{u}$  for some known matrix  $\mathbf{A} \in \mathbb{R}^{m \times n_1}$  and an  $s$ -sparse vector  $\mathbf{u} \in \mathbb{R}^{n_1}$  and  $\mathbf{w} = \mathbf{B}\mathbf{v}$  for some known matrix  $\mathbf{B} \in \mathbb{R}^{m \times n_2}$  and arbitrary vector  $\mathbf{v} \in \mathbb{R}^{n_2}$ , applying the discrete Fourier transform (represented via the DFT matrix  $\mathbf{F} \in \mathbb{C}^{m \times m}$ ), we can write the coordinates of
$$
\mathbf {y} = \mathbf {F} \widetilde {y} = \operatorname {d i a g} (\mathbf {F z}) \mathbf {F w} = \operatorname {d i a g} (\mathbf {F A u}) \mathbf {F B v}
$$
$$
\mathbf {y} _ {j} = \mathcal {A} (\mathbf {u v} ^ {*}) _ {j} = \left\langle \left(\mathbf {F A}\right) _ {j,:} ^ {*} \overline {{\mathbf {F B}}} _ {j,:}, \mathbf {u v} ^ {*} \right\rangle_ {F}
$$
for each  $j = 1, \ldots, m$ , which allows us to write the problem as a simultaneously rank-1 and  $s$ -row sparse recovery problem from Fourier-type measurements.
Figure 5: Left column: RiemAdaIHT, center: SPF, right: IRLS. Success rates for the recovery of low-rank and row-sparse matrices from Fourier rank-one measurements. First row: Rank-1 ground truth  $\mathbf{X}_{\star}$ . Second row: Rank-5 ground truth  $\mathbf{X}_{\star}$  (cf. Figure 2).
In Figure 5, we report the results of simulations with A and B chosen generically as standard real Gaussians for these Fourier-based rank-1 measurements (including for rank-5 ground truths, which goes beyond a blind deconvolution setting). We observe that the transition from no recovery to exact recovery for an increasing number of measurement (with fixed dimension parameters  $s$ ,  $n_1$  and  $n_2$ ) happens earlier than for the random Gaussian rank-one measurements of Appendix A.2, but slightly later than for dense Gaussian measurements. Again, IRLS exhibits the best empirical data-efficiency with sharpest phase transition curves.
As a summary, we observe that IRLS is able to recovery simultaneously low-rank and row-sparse matrices empirically from fewer measurements than state-of-the-art methods for a variety of linear measurement operators, including in cases where the RIP assumption of Definition 2.4 is not satisfied and in cases that are relevant for applications.
A.4 Evolution of Objective Values
While Theorem 2.5 guarantees local convergence if the measurement operator  $\mathcal{A}$  is generic enough and contains enough measurements (RIP-assumption), it is instructive to study the behavior of Algorithm 1 in situations where there are not enough measurements available to identify a specific low-rank and row-sparse ground truth  $\mathbf{X}_{\star}$  which respect to which the measurements have been taken.
In this setting, Theorem 2.6 guarantees that the behavior of the IRLS methods is still benign as the sequence of  $\varepsilon$ - and  $\delta$ -smoothed log-objectives  $\left(\mathcal{F}_{\varepsilon_k,\delta_k}(\mathbf{X}^{(k)})\right)_{k\geq 1}$  from (4) is non-increasing. In Figure 6, we illustrate the evolution of the relative Frobenius error of an iterate to the ground truth  $\mathbf{X}_{\star}$ , the  $(\varepsilon_k,\delta_k)$ -smoothed logarithmic surrogates  $\mathcal{F}_{\varepsilon_k,\delta_k}(\mathbf{X}^{(k)})$  as well as of the rank and row-sparsity parts  $\mathcal{F}_{lr,\varepsilon_k}(\mathbf{X}^{(k)})$  and  $\mathcal{F}_{sp,\delta_k}(\mathbf{X}^{(k)})$  of the objective, respectively, in two typical situations.
In particular, we can see the evolution of these four quantities in the setting of data of dimensionality  $n_1 = 128$ ,  $n_2 \in \{20,40\}$ ,  $s = 20$  and  $r = 5$  created as in the other experiments, where a number of  $m = 875$  and  $m = 175$  (corresponding to an oversampling factor of 3.0 and 1.0, respectively) dense Gaussian measurements are provided to Algorithm 1.
In the left plot of Figure 6, which corresponds to setting of abundant measurements, we observe that the four quantities all track each other relatively well on a semilogarithmic scale (note that we plot the square roots of the objective values to match the order of the (unsquared) relative Frobenius error),
converging to values between  $10^{-13}$  and  $10^{-11}$  (at which point the stopping criterion of the method applies) within 12 iterations.
In the second plot of Figure 6, the number of measurements exactly matches the number of degrees of freedom of the ground truth, in which case the  $\mathbf{X}^{(k)}$  does not converge to  $\mathbf{X}_{\star}$ . However, it can be seen that Algorithm 1 still finds very meaningful solutions: It can be seen that within 86 iterations,  $\mathcal{F}_{\varepsilon_k,\delta_k}(\mathbf{X}^{(k)})$  converges to  $\approx 10^{-12}$  (since  $\sqrt{\mathcal{F}_{\varepsilon_k,\delta_k}(\mathbf{X}^{(k)})} \approx 10^{-6}$ ) in a manner that is partially "staircase-like": After 20 initial iterations where  $\mathcal{F}_{\varepsilon_k,\delta_k}(\mathbf{X}^{(k)})$  decreases significantly at each iteration, its decrease is dominated by relatively sudden, alternating drops of the (blue) sparsity objective  $\mathcal{F}_{sp,\delta_k}(\mathbf{X}^{(k)})$  and the (red) rank objective  $\mathcal{F}_{lr,\varepsilon_k}(\mathbf{X}^{(k)})$ , which typically do not occur simultaneously.
This illustrates the self-balancing property of the two objective terms in the IRLS objective  $\mathcal{F}_{\varepsilon_k,\delta_k}(\mathbf{X}^{(k)})$ : while the final iterate at iteration  $k = 86$  is not of the target row-sparsity  $s = 20$  and  $r = 5$ , it is still 20-row sparse and has essentially rank 6. This means that Algorithm 1 has found an alternative parsimonious solution to the simultaneous low-rank and row-sparse recovery problem that is just slightly less parsimonious.
Arguably, this robust performance in the low-data regime of IRLS is rather unique, and to the best of our knowledge, not shared by methods such as SPF or RiemAdaIHT, which typically breakdown in such a regime.
Figure 6: Objective/ error quantities of iterates  $\mathbf{X}^{(k)}$  for iterations  $k$ . Left: Typical result for  $n_1 = 128$ ,  $n = 40$ ,  $m = 875$ . Right: Typical result for  $n_1 = 128$ ,  $n = 20$ ,  $m = 175$ .
A.5 Robustness under Noisy Measurements
The convergence theory for the IRLS method Algorithm 1 established in Theorem 2.5 assume that exact linear measurements  $\mathbf{y} = \mathcal{A}(\mathbf{X}_{\star})$  of a row-sparse and low-rank ground truth  $\mathbf{X}_{\star}$  are provided to the algorithm. However, in practice, one would expect that the linear measurement model is only approximately accurate. For IRLS for sparse vector recovery, theoretical guarantees have been established for this case in [26, 57]. We do not extend such results to the simultaneously structured case, but we provide numerical evidence that IRLS as defined in Algorithm 1 can be used directly also for noisy measurements.
To this end, we conduct an experiment in the problem setup of Figure 1 in Section 4 for a fixed row-sparsity of  $s = 40$ , in which the measurements provided to the algorithms IRLS, RiemAdaIHT
and SPF are such that
$$
\mathbf {y} = \mathcal {A} (\mathbf {X} _ {\star}) + \mathbf {w},
$$
where  $\mathbf{w}$  is a Gaussian vector (i.i.d. entries) with standard deviation of  $\sigma = \sqrt{\frac{\|\mathcal{A}(\mathbf{X}_{\star})\|_2^2}{m\cdot\mathrm{SNR}}}$  and where SNR is a varying signal-to-noise ratio. We consider SNRs between 10 and  $10^{12}$ , and report the resulting relative Frobenius error statistics in Figure 7.
Figure 7: Median relative Frobenius reconstruction errors of different algorithms given noisy Gaussian measurements,  $n_1 = 256$ ,  $n_2 = 40$ , row-sparsity  $s = 40$  and rank  $r = 1$ , oversampling factor of 3. Error bars correspond to  $25\%$  and  $75\%$  percentiles.
We observe that the reconstruction error is consistently roughly proportional to the inverse square root of the signal-to-noise ratio, for all three algorithms considered. This suggests that IRLS is as noise robust as comparable algorithms, and expected to be return estimates of the original ground truth that has a reconstruction error that is of the order of the norm of the noise.
The following two sections contain the proofs of our main results. Let us begin with some helpful observations.
First note that the low-rank promoting part  $W_{\mathbf{X}^{(k)},\varepsilon_k}^{lr}:\mathbb{R}^{n_1\times n_2}\to \mathbb{R}^{n_1\times n_2}$  of our weight operator can be re-written as
$$
\left. W _ {\mathbf {X} ^ {(k)}, \varepsilon_ {k}} ^ {l r} (\mathbf {Z}) = [ \mathbf {U} \quad \mathbf {U} _ {\perp} ] \left(\mathbf {H} \left(\boldsymbol {\sigma} ^ {(k)}, \varepsilon_ {k}\right) \circ \left(\left[ \begin{array}{l} \mathbf {U} ^ {*} \\ \mathbf {U} _ {\perp} ^ {*} \end{array} \right] \mathbf {Z} [ \mathbf {V} \quad \mathbf {V} _ {\perp} ]\right)\right) \left[ \begin{array}{l} \mathbf {V} ^ {*} \\ \mathbf {V} _ {\perp} ^ {*} \end{array} \right], \right. \tag {18}
$$
$$
\begin{array}{l} \mathbf {H} (\boldsymbol {\sigma} ^ {(k)}, \varepsilon_ {k}) := \left[ \min  \left(\varepsilon_ {k} / \sigma_ {i} ^ {(k)}, 1\right) \min  \left(\varepsilon_ {k} / \sigma_ {j} ^ {(k)}, 1\right) \right] _ {i, j = 1} ^ {n _ {1}, n _ {2}} \\ = \left[ \begin{array}{c c} \left(\frac {\varepsilon_ {k} ^ {2}}{\sigma_ {i} ^ {(k)} \sigma_ {j} ^ {(k)}}\right) _ {i, j = 1} ^ {r _ {k}} & \left(\frac {\varepsilon_ {k}}{\sigma_ {i} ^ {(k)}}\right) _ {i, j = 1} ^ {r _ {k}, d _ {2}} \\ \hline \left(\frac {\varepsilon_ {k}}{\sigma_ {j} ^ {(k)}}\right) _ {i, j = 1} ^ {d _ {1}, r _ {k}} & \mathbf {1} \end{array} \right] \in \mathbb {R} ^ {n _ {1} \times n _ {2}}. \\ \end{array}
$$
Consequently, all weight operators in Definition 2.1 are self-adjoint and positive. Whereas for  $\mathbf{W}_{\mathbf{X}^{(k)},\delta_k}^{sp}$  this is obvious, for  $W_{\mathbf{X}^{(k)},\varepsilon_k}^{lr}$  it follows from the matrix representation
$$
W _ {\mathbf {X} ^ {(k)}, \varepsilon_ {k}} ^ {l r} = \left(\left[ \begin{array}{c c} \mathbf {U} & \mathbf {U} _ {\perp} \end{array} \right] \otimes \left[ \begin{array}{c c} \mathbf {V} & \mathbf {V} _ {\perp} \end{array} \right]\right) \mathbf {D} _ {\mathbf {H} (\boldsymbol {\sigma} ^ {(k)}, \varepsilon_ {k})} \left(\left[ \begin{array}{c c} \mathbf {U} & \mathbf {U} _ {\perp} \end{array} \right] ^ {*} \otimes \left[ \begin{array}{c c} \mathbf {V} & \mathbf {V} _ {\perp} \end{array} \right] ^ {*}\right)
$$
where  $\mathbf{D}_{\mathbf{H}(\pmb{\sigma}^{(k)},\varepsilon_k)} \in \mathbb{R}^{n_1n_2 \times n_1n_2}$  is a diagonal matrix with the entries of  $\mathbf{H}(\pmb{\sigma}^{(k)},\varepsilon_k)$ , which are all positive, on its diagonal.
B.1 Proof of Theorem 2.5
Before approaching the proof of Theorem 2.5, let us collect various important observations. In order to keep the presentation concise, we defer part of the proofs to Appendix C.
For a rank- $r$  matrix  $\mathbf{Z} = \mathbf{U}\boldsymbol{\Sigma}\mathbf{V}^*$ , we define the tangent space of the manifold of rank- $r$  matrices at  $\mathbf{Z}$  as
$$
T _ {\mathbf {U}, \mathbf {V}} := \left\{\mathbf {U} \mathbf {Z} _ {1} ^ {*} + \mathbf {Z} _ {2} \mathbf {V} ^ {*} \colon \mathbf {Z} _ {1} \in \mathbb {R} ^ {n _ {2} \times r}, \mathbf {Z} _ {2} \in \mathbb {R} ^ {n _ {1} \times r} \right\}. \tag {19}
$$
In a similar manner, we can define for  $\mathbf{Z} = \mathbf{U}\pmb{\Sigma}\mathbf{V}^{*}\in \mathcal{M}_{r,s}$  and  $S = \mathrm{supp}(\mathbf{Z}) = \mathrm{supp}(\mathbf{U})\subset [n_1]$  the tangent space of  $\mathcal{M}_r$  restricted to  $S$  as
$$
T _ {\mathbf {U}, \mathbf {V}, S} := \left\{\mathbf {U} \mathbf {Z} _ {1} ^ {*} + \mathbf {Z} _ {2} \mathbf {V} ^ {*}: \mathbf {Z} _ {1} \in \mathbb {R} ^ {n _ {2} \times r}, \mathbf {Z} _ {2} \in \mathbb {R} ^ {n _ {1} \times r} \text {w i t h} \operatorname {s u p p} (\mathbf {Z} _ {2}) = S \right\}. \tag {20}
$$
As the following lemma shows, orthogonal projections onto the sets  $\mathcal{M}_r^{n_1,n_2},\mathcal{N}_s^{n_1,n_2}$ $T_{\mathbf{U},\mathbf{V}}$  , and  $T_{\mathbf{U},\mathbf{V},S}$  can be efficiently computed.
Lemma B.1. We denote the projection operators onto  $\mathcal{M}_r^{n_1,n_2}$  and  $\mathcal{N}_s^{n_1,n_2}$  by  $\mathsf{T}_r$  and  $\mathsf{H}_s$ .  $\mathsf{T}_r$  truncates a matrix to the  $r$  dominant singular values;  $\mathsf{H}_s$  sets all but the  $s$  in  $\ell_2$ -norm largest rows to zero. In case of ambiguities (multiple singular values/rows of same magnitude), by convention we choose the  $r$  (respectively  $s$ ) with smallest index.
For  $\mathbf{U}$  and  $\mathbf{V}$  fixed, the orthogonal projection onto  $T_{\mathbf{U},\mathbf{V}}$  is given by
$$
\mathbb {P} _ {\mathbf {U}, \mathbf {V}} := \mathbb {P} _ {T _ {\mathbf {U}, \mathbf {V}}} \mathbf {Z} = \mathbf {U} \mathbf {U} ^ {*} \mathbf {Z} + \mathbf {Z} \mathbf {V} \mathbf {V} ^ {*} - \mathbf {U} \mathbf {U} ^ {*} \mathbf {Z} \mathbf {V} \mathbf {V} ^ {*}.
$$
For  $S\subset [n_1]$  and  $\mathbf{U},\mathbf{V}$  fixed with  $\mathrm{supp}(\mathbf{U}) = S$  , the orthogonal projection onto  $T_{\mathbf{U},\mathbf{V},S}$  is given by
$$
\begin{array}{l} \mathbb {P} _ {\mathbf {U}, \mathbf {V}, s} := \mathbb {P} _ {T _ {\mathbf {U}, \mathbf {V}, s}} \mathbf {Z} = \mathbb {P} _ {S} \left(\mathbf {U U} ^ {*} \mathbf {Z} + \mathbf {Z V V} ^ {*} - \mathbf {U U} ^ {*} \mathbf {Z V V} ^ {*}\right) \\ = \mathbf {U} \mathbf {U} ^ {*} \mathbf {Z} + \mathbb {P} _ {S} \mathbf {Z} \mathbf {V} \mathbf {V} ^ {*} - \mathbf {U} \mathbf {U} ^ {*} \mathbf {Z} \mathbf {V} \mathbf {V} ^ {*}, \\ \end{array}
$$
where  $\mathbb{P}_S$  projects to the row support  $S$ , i.e., it sets all rows to zero which are not indexed by  $S$ .
The proof of Lemma B.1 is provided in Appendix C.2. In contrast to the above named projections, the projection onto  $\mathcal{M}_{r,s}^{n_1,n_2}$  is not tractable. However, [29, Lemma 2.4] shows that locally  $\mathbb{P}_{\mathcal{M}_{r,s}}$  can be replaced by the concatenation of  $\mathsf{T}_r$  and  $\mathsf{H}_s$ , i.e., for  $\mathbf{Z}_{\star} \in \mathcal{M}_{r,s}$  and  $\mathbf{Z} \approx \mathbf{Z}_{\star}$ , one has that
$$
\mathbb {P} _ {\mathcal {M} _ {r, s}} (\mathbf {Z}) = \mathrm {T} _ {r} \left(\mathrm {H} _ {s} (\mathbf {Z})\right).
$$