SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Remark 2.2. For the sake of conciseness, we only consider row-sparsity here. Algorithm 1 and its analysis can however be modified to cover row- and column-sparse matrices as well. For instance, in the symmetric setting  $\mathbf{X}_{\star} = \mathbf{X}_{\star}^{T}$  (naturally occurring in applications like sparse phase retrieval) one would define the weight operator  $W_{\mathbf{X}^{(k)},\varepsilon_k,\delta_k}$  as in (8), but with an additional term that multiplies  $\mathbf{W}_{\mathbf{X}^{(k)},\delta_k}^{sp}$  from the right to  $\mathbf{Z}$ , which corresponds to minimizing the sum of three smoothed logarithmic surrogates. In this case, the solving modified weighted least squares problem (11) will have similar complexity (potentially smaller complexity, as additional symmetries can be exploited).
Algorithm 1 IRLS for simultaneously low-rank rand row-sparse matrices
1: Input: Linear operator  $\mathcal{A}$ :  $\mathbb{R}^{n_1\times n_2}\to \mathbb{R}^m$ , data  $\mathbf{y}\in \mathbb{R}^m$ , rank and sparsity estimates  $\widetilde{r}$  and  $\widetilde{s}$ .
2: Initialize  $k = 0$ ,  $W_{\mathbf{X}^{(0)}, \varepsilon_0, \delta_0} = \mathbf{Id}$  and set  $\delta_k, \varepsilon_k = \infty$ .
3: for  $k = 1$  to  $K$  do
4: Weighted Least Squares: Update iterate  $\mathbf{X}^{(k)}$  by
$$
\mathbf {X} ^ {(k)} = \underset {\mathbf {X}: \mathcal {A} (\mathbf {X}) = \mathbf {y}} {\arg \min } \langle \mathbf {X}, W _ {\mathbf {X} ^ {(k - 1)}, \varepsilon_ {k - 1}, \delta_ {k - 1}} (\mathbf {X}) \rangle . \tag {11}
$$
5: Update Smoothing: Compute  $\widetilde{r} + 1$ -st singular value  $\sigma_{\widetilde{r} + 1}^{(k)} := \sigma_{\widetilde{r} + 1}(\mathbf{X}^{(k)})$  and  $(\widetilde{s} + 1)$ -st largest row  $\ell_2$ -norm  $\rho_{\widetilde{s} + 1}(\mathbf{X}^{(k)})$  of  $\mathbf{X}^{(k)}$ , update
$$
\varepsilon_ {k} = \min  \left(\varepsilon_ {k - 1}, \sigma_ {\widetilde {r} + 1} ^ {(k)}\right), \quad \delta_ {k} = \min  \left(\delta_ {k - 1}, \rho_ {\widetilde {s} + 1} \left(\mathbf {X} ^ {(k)}\right)\right). \tag {12}
$$
6: Update Weight Operator: For  $r_k$  and  $s_k$  as in (6) and (7),
- compute first  $r_k$  singular triplets  $\sigma^{(k)} \in \mathbb{R}^{r_k}$ ,  $\mathbf{U} \in \mathbb{R}^{n_1 \times r_k}$  and  $\mathbf{V} \in \mathbb{R}^{n_2 \times r_k}$ ,
compute  $W_{\mathbf{X}^{(k)},\varepsilon_k,\delta_k}$  in (8) via  $W_{\mathbf{X}^{(k)},\varepsilon_k}^{lr}$  in (9) and  $\mathbf{W}_{\mathbf{X}^{(k)},\delta_k}^{sp}$  in (10).
8: Output:  $\mathbf{X}^{(K)}$ .
Recall  $\mathcal{F}_{\varepsilon_k,\delta_k},\mathcal{F}_{lr,\varepsilon_k}$ , and  $\mathcal{F}_{sp,\delta_k}$  from (3)-(4). The high-level idea of Algorithm 1, as for other IRLS methods, is to minimize quadratic functionals, which we call  $\mathcal{Q}_{lr,\varepsilon_k}(\cdot |\mathbf{X}^{(k)}):\mathbb{R}^{n_1\times n_2}\to \mathbb{R}$  and  $\mathcal{Q}_{sp,\delta_k}(\cdot |\mathbf{X}^{(k)}):\mathbb{R}^{n_1\times n_2}\to \mathbb{R}$  and define them by
$$
\mathcal {Q} _ {l r, \varepsilon_ {k}} (\mathbf {Z} | \mathbf {X} ^ {(k)}) := \mathcal {F} _ {l r, \varepsilon_ {k}} (\mathbf {X} ^ {(k)}) + \langle \nabla \mathcal {F} _ {l r, \varepsilon_ {k}} (\mathbf {X} ^ {(k)}), \mathbf {Z} - \mathbf {X} ^ {(k)} \rangle + \frac {1}{2} \langle \mathbf {Z} - \mathbf {X} ^ {(k)}, W _ {\mathbf {X} ^ {(k)}, \varepsilon_ {k}} ^ {l r} (\mathbf {Z} - \mathbf {X} ^ {(k)}) \rangle ,
$$
$$
\mathcal {Q} _ {s p, \delta_ {k}} (\mathbf {Z} | \mathbf {X} ^ {(k)}) := \mathcal {F} _ {s p, \delta_ {k}} (\mathbf {X} ^ {(k)}) + \left\langle \nabla \mathcal {F} _ {s p, \delta_ {k}} (\mathbf {X} ^ {(k)}), \mathbf {Z} - \mathbf {X} ^ {(k)} \right\rangle + \frac {1}{2} \left\langle \mathbf {Z} - \mathbf {X} ^ {(k)}, \mathbf {W} _ {\mathbf {X} ^ {(k)}, \delta_ {k}} ^ {s p} (\mathbf {Z} - \mathbf {X} ^ {(k)}) \right\rangle , \tag {13}
$$
that majorize  $\mathcal{F}_{\varepsilon_k,\delta_k}(\cdot)$  (see Theorem 2.6 below) for any iteration  $k$ . This minimization leads to the weighted least squares problem (11) in Algorithm 1. This step can be implemented by standard numerical linear algebra (see the supplementary material for a discussion of its computational complexity). As a second ingredient of the method, the smoothing parameters  $\varepsilon_{k}$  and  $\delta_{k}$  of  $\mathcal{F}_{\varepsilon_k,\delta_k}$  are updated (i.e., decreased) in step (12) before the weight operator is updated according to the current iterate information. For the weight operator update, it is only necessary to compute row norms and leading singular triplets of  $\mathbf{X}^{(k)}$ .
Remark 2.3. The particular form of the low-rank promoting part of the weight operator  $W_{\mathbf{X}^{(k)}, \varepsilon_k}^{lr}$  in (9) is due to [53, 54] and captures optimally spectral information both in the column and row space, unlike prior work on low-rank IRLS [35, 70], while retaining the property that the induced quadratic model  $\mathcal{Q}_{lr, \varepsilon_k}(\cdot | \mathbf{X}^{(k)})$  majorizes  $\mathcal{F}_{lr, \varepsilon_{\varepsilon_k}}(\cdot)$  (see proof of Theorem 2.6). This choice is critical to enable a fast local rate as established in Theorem 2.5.
2.2 Local Quadratic Convergence of IRLS
Our first main result states that Algorithm 1 exhibits quadratic convergence in a local neighborhood of  $\mathbf{X}_{\star}$ , a property Algorithm 1 shares with several methods from the IRLS family. We only need to assume that  $\mathcal{A}$  acts almost isometrically on the set  $\mathcal{M}_{r,s}$ .
Definition 2.4. We say that a linear operator  $\mathcal{A}\colon \mathbb{R}^{n_1\times n_2}\to \mathbb{R}^m$  satisfies the rank-  $r$  and row- $s$ -sparse restricted isometry property (or  $(r,s)$ -RIP) with RIP-constant  $\delta \in (0,1)$  if
$$
(1 - \delta) \| \mathbf {Z} \| _ {F} ^ {2} \leq \| \mathcal {A} (\mathbf {Z}) \| _ {2} ^ {2} \leq (1 + \delta) \| \mathbf {Z} \| _ {F} ^ {2},
$$
for all  $\mathbf{Z} \in \mathcal{M}_{r,s}$ .
It is worth highlighting that Gaussian operators satisfy the above RIP with high-probability if  $m \geq cr(s + n_2) \log (en_1 / s)$ , for some absolute constant  $c > 0$ , see for instance [60]. Up to log-factors, this is at the information theoretic limit which we discussed in the beginning. The convergence result for Algorithm 1 now reads as follows.
Theorem 2.5 (Local Quadratic Convergence). Let  $\mathbf{X}_{\star} \in \mathcal{M}_{r,s}$  be a fixed ground-truth matrix that is  $s$ -row-sparse and of rank  $r$ . Let linear observations  $\mathbf{y} = \mathcal{A}(\mathbf{X}_{\star})$  be given and assume that  $\mathcal{A}$  has the  $(r,s)$ -RIP with  $\delta \in (0,1)$ . Assume that the  $k$ -th iterate  $\mathbf{X}^{(k)}$  of Algorithm 1 with  $\widetilde{r} = r$  and  $\widetilde{s} = s$  updates the smoothing parameters in (12) such that one of the statements  $\varepsilon_{k} = \sigma_{r+1}(\mathbf{X}^{(k)})$  or  $\delta_{k} = \rho_{s+1}(\mathbf{X}^{(k)})$  is true, and that  $r_{k} \geq r$  and  $s_{k} \geq s$ .
If  $\mathbf{X}^{(k)}$  satisfies
$$
\left\| \mathbf {X} ^ {(k)} - \mathbf {X} _ {\star} \right\| \leq \frac {1}{4 8 \sqrt {n} c _ {\| \mathcal {A} \| _ {2 \rightarrow 2}} ^ {3}} \min  \left\{\frac {\sigma_ {r} \left(\mathbf {X} _ {\star}\right)}{r}, \frac {\rho_ {s} \left(\mathbf {X} _ {\star}\right)}{s} \right\} \tag {14}
$$
where  $c_{\| \mathcal{A}\|_{2\to 2}} = \sqrt{1 + \frac{\|\mathcal{A}\|_{2\to 2}^2}{(1 - \delta)}}$  and  $n = \min \{n_1,n_2\}$ , then the local convergence rate is quadratic in the sense that
$$
\| \mathbf {X} ^ {(k + 1)} - \mathbf {X} _ {\star} \| \leq \min  \{\mu \| \mathbf {X} ^ {(k)} - \mathbf {X} _ {\star} \| ^ {2}, 0. 9 \| \mathbf {X} ^ {(k)} - \mathbf {X} _ {\star} \| \},
$$
$$
\mu = 4. 1 7 9 c _ {\| \mathcal {A} \| _ {2} \rightarrow 2} ^ {2} \left(\frac {5 r}{\sigma_ {r} (\mathbf {X} _ {\star})} + \frac {2 s}{\rho_ {s} (\mathbf {X} _ {\star})}\right), \tag {15}
$$
and  $\mathbf{X}^{(k + \ell)}\stackrel {\ell \to \infty}{\rightarrow}\mathbf{X}_{\star}$
The proof of Theorem 2.5 is presented in the supplementary material. To the best of our knowledge, so far no other method exists for recovering simultaneously sparse and low-rank matrices that exhibits local quadratic convergence. In particular, the state-of-the-art competitor methods [60, 37, 69, 29] reach a local linear error decay at best.
On the other hand, (14) is rather pessimistic since for Gaussian  $\mathcal{A}$  the constant  $c_{\| \mathcal{A}\|_{2\to 2}}$  scales like  $\sqrt{(n_1n_2) / m}$ , which means that the right-hand side of (14) behaves like  $m^{3 / 2} / (n(n_1n_2)^{3 / 2})$ , whereas we observe quadratic convergence in experiments within an empirically much larger convergence radius. Closing this gap between theory and practical performance is future work.
It is noteworthy that the theory in [60] — to our knowledge the only other related work explicitly characterizing the convergence radius — holds on a neighborhood of  $\mathbf{X}_{\star}$  that is independent of the ambient dimension. The authors of [60] however assume that the RIP-constant decays with the conditioning number  $\kappa$  of  $\mathcal{A}$ , a quantity that might be large in applications. Hence, ignoring log-factors the sufficient number of measurements in [60] scales like  $m = \Omega (\kappa^{2}r(s_{1} + n_{2}))$ . In contrast, Theorem 2.5 works for any RIP-constant less than one which means for  $m = \Omega (r(s_1 + n_2))$ .
2.3 IRLS as Quadratic Majorize-Minimize Algorithm
With Theorem 2.5, we have provided a local convergence theorem that quantifies the behavior of Algorithm 1 in a small neighbourhood of the simultaneously row-sparse and low-rank ground-truth  $\mathbf{X}_{\star} \in \mathbb{R}^{n_1 \times n_2}$ . The result is based on sufficient regularity of the measurement operator  $\mathcal{A}$ , which in turn is satisfied with high probability if  $\mathcal{A}$  consists of sufficiently generic random linear observations that concentrate around their mean.
In this section we establish that, for any measurement operator  $\mathcal{A}$ , Algorithm 1 can be interpreted within the framework of iteratively reweighted least squares (IRLS) algorithms [26, 70, 75], which implies a strong connection to the minimization of a suitable smoothened objective function. In our case, the objective  $\mathcal{F}_{\varepsilon,\delta}$  in (4) is a linear combination of sum-of-logarithms terms penalizing both non-zero singular values [31, 70, 14] as well as non-zero rows of a matrix  $\mathbf{X}$  [50].
We show in Theorem 2.6 below that the IRLS algorithm Algorithm 1 studied in this paper is based on minimizing at each iteration quadratic models that majorize  $\mathcal{F}_{\varepsilon, \delta}$ , and furthermore, that the iterates  $(\mathbf{X}^{(k)})_{k \geq 1}$  of Algorithm 1 define a non-increasing sequence  $\left(\mathcal{F}_{\varepsilon_k, \delta_k}(\mathbf{X}^{(k)})\right)_{k \geq 1}$  with respect to the objective  $\mathcal{F}_{\varepsilon, \delta}$  of (4). The proof combines the fact that for fixed smoothing parameters  $\varepsilon_k$  and  $\delta_k$ , the weighted least squares and weight update steps Algorithm 1 can be interpreted as a step of a Majorize-Minimize algorithm [86, 58], with a decrease in the underlying objective (4) for updated smoothing parameters.
Theorem 2.6. Let  $\mathbf{y} \in \mathbb{R}^m$ , let the linear operator  $\mathcal{A}\colon \mathbb{R}^{n_1 \times n_2} \to \mathbb{R}^m$  be arbitrary. If  $(\mathbf{X}^{(k)})_{k \geq 1}$  is a sequence of iterates of Algorithm 1 and  $(\delta_k)_{k \geq 1}$  and  $(\varepsilon_k)_{k \geq 1}$  are the sequences of smoothing parameters as defined therein, then the following statements hold.
1. The quadratic model functions  $\mathcal{Q}_{lr,\varepsilon_k}(\cdot |\mathbf{X}^{(k)})$  and  $\mathcal{Q}_{sp,\delta_k}(\cdot |\mathbf{X}^{(k)})$  defined in (13) globally majorize the  $(\varepsilon_{k},\delta_{k})$  -smoothed logarithmic surrogate objective  $\mathcal{F}_{\varepsilon_k,\delta_k}$ , i.e., for any  $\mathbf{Z}\in \mathbb{R}^{n_1\times n_2}$ , it holds that
$$
\mathcal {F} _ {\varepsilon_ {k}, \delta_ {k}} (\mathbf {Z}) \leq \mathcal {Q} _ {l r, \varepsilon_ {k}} \left(\mathbf {Z} \mid \mathbf {X} ^ {(k)}\right) + \mathcal {Q} _ {s p, \delta_ {k}} \left(\mathbf {Z} \mid \mathbf {X} ^ {(k)}\right). \tag {16}
$$
2. The sequence  $\left(\mathcal{F}_{\varepsilon_k,\delta_k}(\mathbf{X}^{(k)})\right)_{k\geq 1}$  is non-increasing.
3. If  $\overline{\varepsilon} := \lim_{k \to \infty} \varepsilon_k > 0$  and  $\bar{\delta} := \lim_{k \to \infty} \delta_k > 0$ , then  $\lim_{k \to \infty} \| \mathbf{X}^{(k)} - \mathbf{X}^{(k+1)} \|_F = 0$ . Furthermore, in this case, every accumulation point of  $(\mathbf{X}^{(k)})_{k \geq 1}$  is a stationary point of
$$
\min  _ {\mathbf {X}: \mathcal {A} (\mathbf {X}) = \mathbf {y}} \mathcal {F} _ {\overline {{\varepsilon}}, \overline {{\delta}}} (\mathbf {X}).
$$
The proof of Theorem 2.6 is presented in the supplementary material.
3 Discussion of Computational Complexity
It is well-known that the solution of the linearly constrained weighted least squares problem (11) can be written as
$$
\mathbf {X} ^ {(k)} = W _ {k - 1} ^ {- 1} \mathcal {A} ^ {*} \left(\mathcal {A} W _ {k - 1} ^ {- 1} \mathcal {A} ^ {*}\right) ^ {- 1} \mathbf {y} \tag {17}
$$
where  $W_{k - 1} \coloneqq W_{\mathbf{X}^{(k - 1)},\varepsilon_{k - 1},\delta_{k - 1}}$  is the weight operator (8) of iteration  $k - 1$  [26, 54]. In [54, Theorem 3.1 and Supplementary Material], it was shown that in the case of low-rank matrix completion without the presence of a row-sparsity inducing term, this weighted least squares problem can be solved by solving an equivalent, well-conditioned linear system via an iterative solver that uses the application of a system matrix whose matrix-vector products have time complexity of  $O(mr + r^2\max (n_1,n_2))$ .
In the case of Algorithm 1, the situations are slightly more involved as we cannot provide an explicit formula for the inverse of the weight operator  $W_{k-1}$  as it amounts to the sum of the weight operators  $W_{\mathbf{X}^{(k-1)}, \varepsilon_{k-1}}^{lr}$  and  $\mathbf{W}_{\mathbf{X}^{(k-1)}, \delta_{k-1}}^{sp}$  that are diagonalized by different, mutually incompatible bases. However, computing this inverse is facilitated by the Sherman-Morrison-Woodbury formula [97]
$$
(\mathbf {E C F} ^ {*} + \mathbf {B}) ^ {- 1} = \mathbf {B} ^ {- 1} - \mathbf {B} ^ {- 1} \mathbf {E} (\mathbf {C} ^ {- 1} + \mathbf {F} ^ {*} \mathbf {B} ^ {- 1} \mathbf {E}) ^ {- 1} \mathbf {F} ^ {*} \mathbf {B} ^ {- 1}
$$
for suitable matrices of compatible size  $\mathbf{E}$ ,  $\mathbf{F}$  and invertible  $\mathbf{C}$ ,  $\mathbf{B}$  and the fact that both  $W_{\mathbf{X}^{(k-1)},\varepsilon_{k-1}}^{lr}$  and  $\mathbf{W}_{\mathbf{X}^{(k-1)},\delta_{k-1}}^{sp}$  exhibit a "low-rank plus (scaled) identity" or a "sparse diagonal plus (scaled) identity" structure. After a simple application of the SMW formula, (17) can be rewritten such that the computational bottleneck becomes the assembly and inversion of a  $O(r_k \max(n_1, n_2))$  linear system. We note that in general, this can be done exactly in a time complexity of  $O(r_k^3 \max(n_1, n_2)^3)$  using standard linear algebra. A crucial factor in the computational cost of the method is also the structure of the measurement operator  $\mathcal{A}$  defining the problem, as the application of itself and its adjoint can significantly influence the per-iteration cost of IRLS; for dense Gaussian measurements, just processing the information of  $\mathcal{A}$  amounts to  $mn_1n_2$  flops. If rank-one or Fourier-type measurements are taken, this cost can significantly be reduced, see [29, Table 1 and Section 3] for an analogous discussion.
We refer to the MATLAB implementation available in the repository https://github.com/ckuemmerle/simirls for further details. While our implementation is not optimized for large-scale problems, the computational cost of Algorithm 1 was observed to be comparable to the implementations of SPF or RiemAdaIHT provided by the authors [69, 29]. We leave further improvements and adaptations to large-scale settings to future work.
4 Numerical Evaluation
In this section, we explore the empirical performance of IRLS in view of the theoretical results of Theorems 2.5 and 2.6, and compare its ability to recover simultaneous low-rank and row-sparse data matrices with the state-of-the-art methods Sparse Power Factorization (SPF) [60] and Riemannian adaptive iterative hard thresholding (RiemAdaIHT) [29], which are
among the methods with the best empirical performance reported in the literature. The method ATLAS [33] and its successor [69] are not used in our empirical studies since they are tailored to robust recovery and yield suboptimal performance when seeking high-precision reconstruction in low noise scenarios. We use spectral initialization for SPF and RiemAdaIHT. The weight operator of IRLS is initialized by the identity as described in Algorithm 1, solving an unweighted least squares problem in the first iteration. A detailed description of the experimental setup can be found in Appendix A.1.
Performance in Low-Measurement Regime. Figures 1 and 2 show the empirical probability of successful recovery when recovering  $s$ -row sparse ground-truths  $\mathbf{X}_{\star} \in \mathbb{R}^{256 \times 40}$  of rank  $r = 1$  (resp.  $r = 5$ ) from Gaussian measurements under oracle knowledge on  $r$  and  $s$ . The results are averaged over 64 random trials. As both figures illustrate, the region of success of IRLS comes closest to the information theoretic limit of  $r(s + n_2 - r)$  which is highlighted by a red line, requiring a significantly lower oversampling factor than the baseline methods.
Figure 1: Left column: RiemAdaIHT, center: SPF, right: IRLS. Phase transition experiments with  $n_1 = 256$ ,  $n_2 = 40$ ,  $r = 1$ , Gaussian measurements. Algorithmic hyperparameters informed by model order knowledge (i.e.,  $\widetilde{r} = r$  and  $\widetilde{s} = s$  for IRLS). White corresponds to empirical success rate of 1, black to 0.
Figure 2: Left column: RiemAdaIHT, center: SPF, right: IRLS. First row: As in Figure 1, but for data matrix  $\mathbf{X}_{\star}$  of rank  $r = 5$ . Second row: As first row, but hyper-parameters  $r$  and  $s$  are overestimated as  $\widetilde{r} = 2r = 10$ ,  $\widetilde{s} = \lfloor 1.5s\rfloor$
In Appendix A.2 and Appendix A.3 in the supplementary material, we report on similar experiments conducted for other measurement operators than dense Gaussians, in which cases the empirical relative behavior of the methods is comparable.
Sensitivity to Parameter Choice. In applications of our setting, the quantities  $r$  and  $s$  might be unknown or difficult to estimate. In the second row of Figure 2, we repeat the experiment of the first row (rank-5 ground truth), but run the algorithms with rank and sparsity estimates of  $\widetilde{r} = 2r$  and  $\widetilde{s} = \lfloor 1.5s\rfloor$ . Whereas all considered methods suffer a deterioration of performance, we observe that IRLS deteriorates relatively the least by a large margin. Furthermore, we observe that even if IRLS does not recovery  $\mathbf{X}_{\star}$ , it converges typically to a matrix that is still low-rank and row-sparse (with
larger  $r$  and  $s$ ) satisfying the data constraint, while the other methods fail to converge to such a matrix.
Convergence Behavior. Finally, we examine the convergence rate of the iterates to validate the theoretical prediction of Theorem 2.5 in the setting of Figure 2. Figure 3 depicts in log-scale the approximation error over the iterates of SPF, RiemAdaIHT, and IRLS. We observe that the IRLS indeed exhibits empirical quadratic convergence within a few iterations (around 10), whereas the other methods clearly only exhibit linear convergence. The experiment further suggests that the rather pessimistic size of the convergence radius established by Theorem 2.5 could possibly be improved by future investigations.
Figure 3: Comparison of convergence rate. Setting as in Figure 2 with  $s = 40$  and  $m = 1125$ .
Further experiments. In Appendix A.4, we provide additional experiments investigating the selfbalancing property of the objective (4), as well as the experiments on the noise robustness of the method in Appendix A.5.
5 Conclusion, Limitations and Future Work
Conclusion. In this paper, we adapted the IRLS framework to the problem of recovering simultaneously structured matrices from linear observations focusing on the special case of row sparsity and low-rankness. Our convergence guarantee Theorem 2.5 is hereby the first one for any method minimizing combinations of structural surrogate objectives that holds in the information-theoretic near-optimal regime and exhibits local quadratic convergence. The numerical experiments we conducted for synthetic data suggest that, due to its weak dependence on the choice of hyperparameters, IRLS in the form of Algorithm 1 can be a practical method for identifying simultaneously structured data even in difficult problem instances.
Limitations and Future Work. As in the case of established IRLS methods that optimize nonconvex surrogate objectives representing a single structure [26, 56, 54, 75], the radius of guaranteed quadratic convergence in Theorem 2.5 is the most restrictive assumption. Beyond the interpretation in terms of surrogate minimization as presented in Theorem 2.6, which holds without any assumptions on the initialization, our method shares a lack of a global convergence guarantees with other non-convex IRLS algorithms [70, 54, 75].
The generalization and application of our framework to combinations of structures beyond rank and row- (or column-)sparsity lies outside the scope of the present paper, but could involve subspace-structured low-rankness [31, 24, 99, 87] or analysis sparsity [30, 77]. A generalization of the presented IRLS framework to higher-order objects such as low-rank tensors is of future interest as convexifications of structure-promoting objectives face similar challenges [71, 74, 98] in this case.
In parameter-efficient deep learning, both sparse [39, 44, 32, 91] and low-rank [96, 94, 82] weight parameter models have gained considerable attention due to the challenges of training and storing, e.g., in large transformer-based models [92, 9]. It will be of interest to study whether in this non-linear case IRLS-like preconditioning of the parameter space can find network weights that are simultaneously sparse and low-rank, and could potentially lead to further increases in efficiency.
The authors thank Max Pfeffer for providing their implementation of the algorithm RiemAdaIHT as considered in [29].