SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Christian Kümmerle
Charlotte, NC 28223, USA
kuemmerle@uncc.edu
Johannes Maly
80799 Munich, Germany and
Munich Center for Machine Learning (MCML)
maly@math.lmu.de
We propose a new algorithm for the problem of recovering data that adheres to multiple, heterogeneous low-dimensional structures from linear observations. Focusing on data matrices that are simultaneously row-sparse and low-rank, we propose and analyze an iteratively reweighted least squares (IRLS) algorithm that is able to leverage both structures. In particular, it optimizes a combination of non-convex surrogates for row-sparsity and rank, a balancing of which is built into the algorithm. We prove locally quadratic convergence of the iterates to a simultaneously structured data matrix in a regime of minimal sample complexity (up to constants and a logarithmic factor), which is known to be impossible for a combination of convex surrogates. In experiments, we show that the IRLS method exhibits favorable empirical convergence, identifying simultaneously row-sparse and low-rank matrices from fewer measurements than state-of-the-art methods.
Reconstructing an image from (noisy) linear observations is maybe the most relevant inverse problem for modern image processing and appears in various applications like medical imaging and astronomy [7]. If the latent image is  $n$ -dimensional, for  $n \in \mathbb{N}$ , it is well-known that  $\Omega(n)$  observations are required for robust identification in general. In practice, imaging problems are however often ill-posed, i.e., the number of observations is smaller than  $n$  or the operator creating the observations is defective [88, 93]. In such situations, the fundamental lower bound of  $\Omega(n)$  can be relaxed by leveraging structural priors of the latent image in the reconstruction process.
Of the various priors that are used for solving ill-posed inverse problems in the literature, sparsity<sup>1</sup> and low-rankness are most prevalent. This prominent role can be explained with their competitive performance in imaging tasks and the rigorous mathematical analysis they allow [10, 68]. For instance, consider the recovery of an  $n_1 \times n_2$ -dimensional image  $\mathbf{X}_{\star} \in \mathbb{R}^{n_1 \times n_2}$  from linear observations
$$
\mathbf {y} = \mathcal {A} \left(\mathbf {X} _ {\star}\right) + \boldsymbol {\eta} \in \mathbb {R} ^ {m}, \tag {1}
$$
where  $\mathcal{A}\colon \mathbb{R}^{n_1\times n_2}\to \mathbb{R}^m$  is a linear operator modeling the impulse response of the sensing device and  $\pmb {\eta}\in \mathbb{R}^m$  models additive noise. Whereas this problem is ill-posed for  $m < n_{1}n_{2}$ , it has been established [38, 15, 78] that it becomes well-posed if  $\mathbf{X}_{\star}$  is sparse or of low rank. The aforementioned works prove that  $m = \Omega (s_1s_2)$  observations suffice for robust reconstruction if  $\mathbf{X}_{\star}$  is  $s_1$ -row-sparse
<sup>1</sup>A vector  $\mathbf{x} \in \mathbb{R}^n$  is called  $s$ -sparse if  $\mathbf{x}$  has at most  $s$  non-zero entries. For a matrix  $\mathbf{X} \in \mathbb{R}^{n_1 \times n_2}$  there are various ways to count the level of sparsity. In this work, we use the most common definition and call  $\mathbf{X}$ $s$ -row-sparse (resp. -column-sparse) if at most  $s$  rows (resp. columns) are non-zero.
and  $s_2$ -column-sparse, and that  $m = \Omega(r(n_1 + n_2))$  observations suffice if  $\mathbf{X}_{\star}$  is a rank- $r$  matrix. These bounds, which relax the general lower bound of  $m = \Omega(n_1n_2)$ , agree with the degrees of freedom of sparse and low-rank matrices, respectively.
A number of computationally challenging problems in signal processing and machine learning can be formulated as instances of (1) with  $\mathbf{X}_{\star}$  being simultaneously structured, i.e.,  $\mathbf{X}_{\star}$  is both of rank  $r$  and  $s_1$ -row-sparse/s2-column-sparse. Examples encompass sparse phase retrieval [51, 46, 11, 47, 85], sparse blind deconvolution [59, 83], hyperspectral imaging [43, 41, 89, 102], sparse reduced-rank regression [22, 100], and graph denoising and refinement [80, 103]. In these settings, the hope is that due to leveraging the simultaneous structure,  $\Omega(r(s_1 + s_2))$  observations suffice to identify the data matrix. For  $r \ll s_1, s_2 \ll n_1, n_2$ , these bounds are significantly smaller than the bounds for single-structured data matrices.
From an algorithmic point of view, however, the simultaneously structured recovery problem poses obstacles that are not present for problems where  $\mathbf{X}_{\star}$  is only of low-rank or (group) sparse: In the latter case, variational methods [6] that formulate the reconstruction method in terms of optimizing a suitable objective with a structural regularization term involving  $\ell_p / \ell_{2,p}$ -(quasi-)norms and and  $S_{p}$ -Schatten (quasi-)norms have been well-understood, leading to tractable algorithms in the information theoretically optimal regime [28, 17, 38].
For simultaneously structured problems, on the other hand, Oymak et al. showed in [74] that a mere linear combination of convex regularizers for different sparsity structures — in our case, nuclear and  $\ell_{2,1}$ -norms — cannot outperform recovery guarantees of the "best" one of them alone. While this indicates that leveraging two priors at once is a way more intricate problem than leveraging a single prior, it was also shown in [74] that minimizing a linear combination of rank and row sparsity can indeed lead to guaranteed recovery from  $\Omega(r(s_1 + s_2))$  measurements. The downside is that the combination of these non-convex and discontinuous quantities does not lend itself directly to practical optimization algorithms, and to the best of our knowledge, so far, there have been no works directly tackling the optimization of a combination of non-convex surrogates that come with any sort of convergence guarantees.
In this work, we approach the reconstruction of simultaneously sparse and low-rank matrices by leveraging the positive results of [74] for non-convex regularizers. To this end, we introduce a family of non-convex, but continuously differentiable regularizers that are tailored to the recovery problem for simultaneously structured data. The resulting objectives lend themselves to efficient optimization by a novel algorithm from the class of iteratively reweighted least squares (IRLS) [26, 35, 70, 5, 1, 54, 55], the convergence of which we analyze in the information theoretically (near-)optimal regime. Specifically, our main contributions are threefold:
(i) In Algorithm 1, we propose a novel IRLS method that is tailored to leveraging both structures of the latent solution, sparsity and low-rankness, at once. The core components of the algorithm are the weight operator defined in Definition 2.1 and the update of the smoothing parameters in (12). Notably, the algorithm automatically balances between its low-rank and its sparsity promoting terms, leading to a reliable identification of  $s_1$ -row-sparse and rank- $r$  ground truths<sup>2</sup>.
(ii) Under the assumption that  $\mathcal{A}$  behaves almost isometrically on the set of row-sparse and low-rank matrices, we show in Theorem 2.5 that locally Algorithm 1 exhibits quadratic convergence towards  $\mathbf{X}_{\star}$ . Note that if  $\mathcal{A}$  is, e.g., a Gaussian operator, the isometry assumption is (up to log factors) fulfilled in the information theoretic (near-)optimal regime  $m = \Omega(r(s_1 + n_2))$  [60].
(iii) Finally, in Section 2.3 we identify the underlying family of objectives that are minimized by Algorithm 1. To make this precise, we define for  $\tau >0$  and  $e$  denoting Euler's number the real-valued function  $f_{\tau}:\mathbb{R}\to \mathbb{R}$  such that
$$
f _ {\tau} (t) = \left\{ \begin{array}{l l} \frac {1}{2} \tau^ {2} \log \left(e t ^ {2} / \tau^ {2}\right), & \text {i f} | t | > \tau , \\ \frac {1}{2} t ^ {2}, & \text {i f} | t | \leq \tau , \end{array} \right. \tag {2}
$$
To enhance readability of the presented proofs and results, we restrict ourselves to row-sparsity of  $\mathbf{X}_{\star}$  here. It is straightforward to generalize the arguments to column-sparsity as well.
which is quadratic around the origin and otherwise logarithmic in its argument. Using this definition, we define for  $\varepsilon > 0$  the  $(\varepsilon-)$ smoothed log-determinant objective  $\mathcal{F}_{lr,\varepsilon} : \mathbb{R}^{n_1 \times n_2} \to \mathbb{R}$  and for  $\delta > 0$  the  $(\delta-)$ smoothed sum of logarithmic row-wise  $\ell_2$ -norms objective  $\mathcal{F}_{sp,\delta} : \mathbb{R}^{n_1 \times n_2} \to \mathbb{R}$  such that
$$
\mathcal {F} _ {l r, \varepsilon} (\mathbf {X}) = \sum_ {r = 1} ^ {\min  \left\{n _ {1}, n _ {2} \right\}} f _ {\varepsilon} \left(\sigma_ {r} (\mathbf {X})\right), \quad \mathcal {F} _ {s p, \delta} (\mathbf {X}) = \sum_ {i = 1} ^ {n _ {1}} f _ {\delta} \left(\left\| \mathbf {X} _ {i,:} \right\| _ {2}\right). \tag {3}
$$
Combining the above, we further define the  $(\varepsilon, \delta-)$  smoothed logarithmic surrogate objective  $\mathcal{F}_{\varepsilon, \delta}: \mathbb{R}^{n_1 \times n_2} \to \mathbb{R}$  as
$$
\mathcal {F} _ {\varepsilon , \delta} (\mathbf {X}) := \mathcal {F} _ {l r, \varepsilon} (\mathbf {X}) + \mathcal {F} _ {s p, \delta} (\mathbf {X}). \tag {4}
$$
In Theorem 2.6, we prove for any  $\mathcal{A}$  that the iterates of Algorithm 1 minimize quadratic majorizations of  $\mathcal{F}_{\varepsilon,\delta}$  and form a non-increasing sequence on  $\mathcal{F}_{\varepsilon,\delta}$ . To the best of our knowledge, the proposed method is the so far only approach for recovering simultaneously sparse and low-rank matrices which combines local (quadratic) convergence with a rigorous variational interpretation.
The numerical simulations in Section 4 support our theoretical findings and provide empirical evidence for the efficacy of the proposed method.
Sparse and Low-Rank Recovery. Whereas leveraging a single matrix structure like sparsity or low-rankness in the reconstruction process can easily be obtained by convex regularizers [78, 16], Oymak et al. [74] showed that, if one is interested in near-optimal sampling rates, one cannot expect comparably simple solutions for identifying simultaneously structured objects; a minimization of (4) with convex terms  $\mathcal{F}_{lr,\varepsilon}(\cdot)$  and  $\mathcal{F}_{sp,\delta}(\cdot)$  would be only as good as using the one structure that is information theoretically more favorable. A closely related problem<sup>3</sup> that appears in statistical literature under the name Sparse Principal Component Analysis (SPCA) [104, 25] is known to be NP-hard in general [67]. Despite the intrinsic hardness of simultaneously structured recovery problems, promising empirical results for hyperspectral image demixing were shown in [40], where minimization problems involving the sum of reweighted convex surrogates are solved by a proximal scheme based on ADMM for the case of simultaneously sparse, low-rank and non-negative matrices. For the problem of simultaneously sparse and low-rank matrix recovery, there exist only a handful approaches that come with rigorous theoretical analysis. The first line of works [4, 37] aims to overcome the aforementioned limitations of purely convex methods in a neat way. They assume that the operator  $\mathcal{A}$  has a nested structure such that basic solvers for low-rank resp. row/column-sparse recovery can be applied in two consecutive steps. Despite being an elegant idea, this approach clearly restricts possible choices for  $\mathcal{A}$  and is of hardly any practical use.
In a second line of work, Lee et al. [60] consider general impulse response operators that satisfy a suitable restricted isometry property for  $s_1$ -row- and  $s_2$ -column-sparse rank- $r$  matrices. They propose and analyze a highly efficient, greedy method, the so-called Sparse Power Factorization (SPF) which is a modified version of power factorization [48] and uses hard thresholding pursuit [36] to enforce sparsity in addition. In particular, they show that if  $\mathbf{X}_{\star}$  is rank- $R$ , has  $s_1$ -sparse columns and  $s_2$ -sparse rows, then  $m \gtrsim R(s_1 + s_2) \log(\max \{en_1 / s_1, en_2 / s_2\})$ . Gaussian observations suffice for robust recovery, which is up to the log-factor at the information theoretical limit we discussed above. The result however assumes a low noise level and requires that SPF is initialized by a, in general, intractable method. Only in the special case that  $\mathbf{X}_{\star}$  is spiky, which means that the norms of non-zero rows/columns exhibit a fast decay, a tractable substitute for the initialization method works provably. The analysis of SPF has been extended to the blind deconvolution setup in [59]. In [100], a related approach that combines gradient descent of a smooth objective with hard thresholding is considered, for which the authors show linear convergence from a suitable initialization if the measurement operator satisfies a restricted strong convexity and smoothness assumption.
A third line of work, approaches the problem from a variational point of view. In [33, 69] the
<sup>3</sup>Since observations in SPCA are provided from noisy samples of the underlying distribution, whereas in our case the matrix itself is observed indirectly, it is however hard to directly compare results from sparse and low-rank matrix reconstruction with corresponding results for SPCA.
authors aim at enhancing robustness of recovery by alternating minimization of an  $\ell_1$ -norm based multi-penalty functional. In essence, the theoretical results bound the reconstruction error of global minimizers of the proposed functional depending on the number of observations. Although the authors only provide local convergence guarantees for the proposed alternating methods, the theoretical error bounds for global minimizers hold for arbitrarily large noise magnitudes and a wider class of ground-truth matrices than the one considered in [60].
The works [37, 29], which build upon generalized projection operators to modify iterative hard thresholding to the simultaneous setting, share the lack of global convergence guarantees.
In [79], the authors examine the use of atomic norms to perform recovery of simultaneously sparse and low-rank matrices, which uses a related, but different sparsity assumption compared to the row or column sparsity studied here. From a practical point of view, the such norms are hard to compute and the paper only proposes a heuristic polynomial time algorithm for the problem.
Finally, the alternative approach of using optimally weighted sums or maxima of convex regularizers [52] requires optimal tuning of the parameters under knowledge of the ground-truth.
Iteratively Reweighted Least Squares. The herein proposed iteratively reweighted least squares algorithm builds on a long line of research on IRLS going back to Weiszfeld's algorithm proposed in the 1930s for a facility location problem [95, 5]. IRLS is a practical framework for the optimization of non-smooth, possibly non-convex, high-dimensional objectives that minimizes quadratic models which majorize these objectives. Due to its ease of implementation and favorable data-efficiency, it has been widely used in compressed sensing [42, 18, 26, 57, 34, 55], robust statistics [45, 2, 72], computer vision [19, 61, 84], low-rank matrix recovery and completion [35, 70, 56, 54], and in inverse problems involving group sparsity [21, 101, 20]. Recently, it has been shown [62] that dictionary learning techniques can be incorporated into IRLS schemes for sparse and low-rank recovery to allow the learning of a sparsifying dictionary while recovering the solution. Whereas IRLS can be considered as a type of majorize-minimize algorithm [58], optimal performance is achieved if intertwined with a smoothing strategy for the original objective, in which case globally linear (for convex objectives) [26, 1, 72, 55, 75] and locally superlinear (for non-convex objectives) [26, 56, 54, 75] convergence rates have been shown under suitable conditions on the linear operator  $\mathcal{A}$ .
However, there has only been little work on IRLS optimizing a sum of heterogenous objectives [81] — including the combination of low-rank promoting and sparsity-promoting objectives — nor on the convergence analysis of any such methods. The sole algorithmically related approach for our setting has been studied in [23], where a method has been derived in a sparse Bayesian learning framework, the main step of which amounts to the minimization of weighted least squares problems. Whereas the algorithm of [23] showcases that such a method can empirically identify simultaneously structured matrices from a small number of measurements, no convergence guarantees or rates have been provided in the information-theoretically optimal regime. Furthermore, [23] only focuses on general sparsity rather than row or column sparsity.
We denote matrices and vectors by bold upper- and lower-case letters to distinguish them from scalars and functions. We furthermore denote the  $i$ -th row of a matrix  $\mathbf{Z} \in \mathbb{R}^{n_1 \times n_2}$  by  $\mathbf{Z}_{i,j}$  and the  $j$ -th column of  $\mathbf{Z}$  by  $\mathbf{Z}_{:j}$ . We abbreviate  $n = \min\{n_1, n_2\}$ . We denote the  $r$ -th singular value of a matrix  $\mathbf{Z} \in \mathbb{R}^{n_1 \times n_2}$  by  $\sigma_r(\mathbf{Z})$ . Likewise, we denote the in  $\ell_2$ -norm  $s$ -largest row of  $\mathbf{Z}$  by  $\rho_s(\mathbf{Z})$ . (To determine  $\rho_s(\mathbf{Z})$ , we form the in  $\ell_2$ -norm non-increasing rearrangement of the rows of  $\mathbf{Z}$  and, by convention, sort rows with equal norm according to the row-index.) We use  $\circ$  to denote the Hadamard (or Schur) product, i.e., the entry-wise product of two vectors/matrices.
We denote the Euclidean  $\ell_2$ -norm of a vector  $\mathbf{z} \in \mathbb{R}^n$  by  $\| \mathbf{z} \|_2$ . For  $\mathbf{Z} \in \mathbb{R}^{n_1 \times n_2}$ , the matrix norms we use encompass the operator norm  $\| \mathbf{Z} \| := \sup_{\| \mathbf{W} \|_2 = 1} \| \mathbf{Z} \mathbf{w} \|_2$ , the row-sum  $p$ -quasinorm  $\| \mathbf{Z} \|_{p,2} := \left( \sum_{i=1}^{n_1} \| \mathbf{Z}_{i,:} \|_2^p \right)^{1/p}$ , the row-max norm  $\| \mathbf{Z} \|_{\infty,2} := \max_{i \in [n_1]} \| \mathbf{Z}_{i,:} \|_2$ , and the Schatten-p quasinorm  $\| \mathbf{Z} \|_{S_p} := \left( \sum_{r=1}^{n} \sigma_r (\mathbf{Z})^p \right)^{1/p}$ . Note that two special cases of Schatten quasinorms are the nuclear norm  $\| \mathbf{Z} \|_* := \| \mathbf{Z} \|_{S_1}$  and the Frobenius norm  $\| \mathbf{Z} \|_F := \| \mathbf{Z} \|_{S_2}$ .
2 IRLS for Sparse and Low-Rank Reconstruction
Recall that we are interested in recovering a rank- $r$  and  $s$ -row-sparse matrix  $\mathbf{X}_{\star} \in \mathbb{R}^{n_1 \times n_2}$  from  $m$  linear observations
$$
\mathbf {y} = \mathcal {A} \left(\mathbf {X} _ {\star}\right) \in \mathbb {R} ^ {m}, \tag {5}
$$
i.e.,  $\mathcal{A}\colon \mathbb{R}^{n_1\times n_2}\to \mathbb{R}^m$  is linear. We write  $\mathbf{X}_{\star}\in \mathcal{M}_{r,s}^{n_{1},n_{2}}\coloneqq \mathcal{M}_{r}^{n_{1},n_{2}}\cap \mathcal{N}_{s}^{n_{1},n_{2}}$ , where  $\mathcal{M}_r^{n_1,n_2}\subset$ $\mathbb{R}^{n_1\times n_2}$  denotes the set of matrices with rank at most  $r$  and  $\mathcal{N}_s^{n_1,n_2}\subset \mathbb{R}^{n_1\times n_2}$  denotes the set of matrices with at most  $s$  non-zero rows. For convenience, we suppress the indices  $n_1$  and  $n_2$  whenever the ambient dimension is clear from the context. In particular, we know that  $\mathbf{X}_{\star} = \mathbf{U}_{\star}\pmb{\Sigma}_{\star}\mathbf{V}_{\star}^{*}$ , where  $\mathbf{U}_{\star}\in \mathcal{N}_s^{n_1,r}$ ,  $\pmb{\Sigma}_{\star}\in \mathbb{R}^{r\times r}$ , and  $\mathbf{V}_{\star}\in \mathbb{R}^{n_2\times r}$  denote the reduced SVD of  $\mathbf{X}_{\star}$ . Furthermore, the row supports of  $\mathbf{X}_{\star}$  and  $\mathbf{U}_{\star}$  (the index sets of non-zero rows of  $\mathbf{X}_{\star}$  resp.  $\mathbf{U}_{\star}$ ) are identical, i.e.,  $\mathrm{supp}(\mathbf{X}_{\star}) = \mathrm{supp}(\mathbf{U}_{\star}) = S_{\star}\subset [n_1]\coloneqq \{1,\dots ,n_1\}$ .
2.1 How to Combine Sparse and Low-Rank Weighting
As discussed in Section 1, the challenge in designing a reconstruction method for (5) lies in simultaneously leveraging both structures of  $\mathbf{X}_{\star}$  in order to achieve the optimal sample complexity of  $m \approx r(s + n_2)$ . To this end, we propose a novel IRLS-based approach in Algorithm 1. The key ingredient for computing the  $(k + 1)$ -st iterate  $\mathbf{X}^{(k + 1)} \in \mathbb{R}^{n_1 \times n_2}$  in Algorithm 1 is the multi-structural weight operator  $W_{\mathbf{X}^{(k)}, \varepsilon_k, \delta_k}: \mathbb{R}^{n_1 \times n_2} \to \mathbb{R}^{n_1 \times n_2}$  of Definition 2.1, which depends on the current iterate  $\mathbf{X}^{(k)}$ .
Definition 2.1. For  $\delta_k, \varepsilon_k > 0$ ,  $\sigma_i^k \coloneqq \sigma_i(\mathbf{X}^{(k)})$ , and  $\mathbf{X}^{(k)} \in \mathbb{R}^{n_1 \times n_2}$ , let
$$
r _ {k} := \left| \left\{i \in [ n ]: \sigma_ {i} ^ {(k)} > \varepsilon_ {k} \right\} \right| \tag {6}
$$
denote the number of singular values  $\sigma^{(k)} = \left(\sigma_{i}^{(k)}\right)_{i = 1}^{r_{k}}$  of  $\mathbf{X}^{(k)}$  larger than  $\varepsilon_{k}$  and furthermore be
$$
s _ {k} := \left| \left\{i \in [ n _ {1} ]: \| \mathbf {X} _ {i,:} ^ {(k)} \| _ {2} > \delta_ {k} \right\} \right| \tag {7}
$$
the number of rows of  $\mathbf{X}^{(k)}$  with  $\ell_2$ -norm larger than  $\delta_k$ . Define the  $r_k$  left and right singular vectors of  $\mathbf{X}^{(k)}$  as columns of  $\mathbf{U} \in R^{n_1 \times r_k}$  and  $\mathbf{V} \in \mathbb{R}^{n_2 \times r_k}$ , respectively, corresponding to the leading singular values  $\sigma^{(k)} = (\sigma_i^{(k)})_{i=1}^{r_k} := (\sigma_i(\mathbf{X}^{(k)}))_{i=1}^{r_k}$ .
We define the weight operator  $W_{\mathbf{X}^{(k)},\varepsilon_k,\delta_k}:\mathbb{R}^{n_1\times n_2}\to \mathbb{R}^{n_1\times n_2}$  at iteration  $k$  of Algorithm 1 as
$$
W _ {\mathbf {X} ^ {(k)}, \varepsilon_ {k}, \delta_ {k}} (\mathbf {Z}) = W _ {\mathbf {X} ^ {(k)}, \varepsilon_ {k}} ^ {l r} (\mathbf {Z}) + \mathbf {W} _ {\mathbf {X} ^ {(k)}, \delta_ {k}} ^ {s p} \mathbf {Z} \tag {8}
$$
where  $W_{\mathbf{X}^{(k)},\varepsilon_k}^{lr}:\mathbb{R}^{n_1\times n_2}\to \mathbb{R}^{n_1\times n_2}$  is its low-rank promoting part
$$
W _ {\mathbf {X} ^ {(k)}, \varepsilon_ {k}} ^ {l r} (\mathbf {Z}) = \left[ \begin{array}{l l} \mathbf {U} & \mathbf {U} _ {\perp} \end{array} \right] \boldsymbol {\Sigma} _ {\varepsilon_ {k}} ^ {- 1} \left[ \begin{array}{l} \mathbf {U} ^ {*} \\ \mathbf {U} _ {\perp} ^ {*} \end{array} \right] \mathbf {Z} \left[ \begin{array}{l l} \mathbf {V} & \mathbf {V} _ {\perp} \end{array} \right] \boldsymbol {\Sigma} _ {\varepsilon_ {k}} ^ {- 1} \left[ \begin{array}{l} \mathbf {V} ^ {*} \\ \mathbf {V} _ {\perp} ^ {*} \end{array} \right] \tag {9}
$$
with  $\mathbf{\Sigma}_{\varepsilon_k} = \max (\sigma_i^{(k)} / \varepsilon_k,1)$  and  $\mathbf{W}_{\mathbf{X}^{(k)},\delta_k}^{sp}\in \mathbb{R}^{n_1\times n_1}$  is its sparsity-promoting part, which is diagonal with
$$
\left(\mathbf {W} _ {\mathbf {X} ^ {(k)}, \delta_ {k}} ^ {s p}\right) _ {i i} = \max  \left(\left\| \left(\mathbf {X} ^ {(k)}\right) _ {i,:} \right\| _ {2} ^ {2} / \delta_ {k} ^ {2}, 1\right) ^ {- 1}, \quad \text {f o r a l l} i \in [ n _ {1} ]. \tag {10}
$$
The matrices  $\mathbf{U}_{\perp}$  and  $\mathbf{V}_{\perp}$  are arbitrary complementary orthogonal bases for  $\mathbf{U}$  and  $\mathbf{V}$  that do not need to be computed in Algorithm 1.