Title: Recovering Simultaneously Structured Data via Non-Convex Iteratively Reweighted Least Squares
Venue: NeurIPS 2023
Accepted: yes
Keywords: low-rank models

sparsity

iteratively reweighted least squares

non-convex optimization

quadratic convergence

simultaneously structured data
Abstract: We propose a new algorithm for the problem of recovering data that adheres to multiple, heterogenous low-dimensional structures from linear observations. Focussing on data matrices that are simultaneously row-sparse and low-rank, we propose and analyze an iteratively reweighted least squares (IRLS) algorithm that is able to leverage both structures. In particular, it optimizes a combination of non-convex surrogates for row-sparsity and rank, a balancing of which is built into the algorithm. We prove locally quadratic convergence of the iterates to a simultaneously structured data matrix in a regime of minimal sample complexity (up to constants and a logarithmic factor), which is known to be impossible for a combination of convex surrogates. In experiments, we show that the IRLS method exhibits favorable empirical convergence, identifying simultaneously row-sparse and low-rank matrices from fewer measurements than state-of-the-art methods.

Recovering Simultaneously Structured Data via Non-Convex Iteratively Reweighted Least Squares

Abstract

We propose a new algorithm for the problem of recovering data that adheres to multiple, heterogeneous low-dimensional structures from linear observations. Focusing on data matrices that are simultaneously row-sparse and low-rank, we propose and analyze an iteratively reweighted least squares ( IRLS ) algorithm that is able to leverage both structures. In particular, it optimizes a combination of non-convex surrogates for row-sparsity and rank, a balancing of which is built into the algorithm. We prove locally quadratic convergence of the iterates to a simultaneously structured data matrix in a regime of minimal sample complexity (up to constants and a logarithmic factor), which is known to be impossible for a combination of convex surrogates. In experiments, we show that the IRLS method exhibits favorable empirical convergence, identifying simultaneously row-sparse and low-rank matrices from fewer measurements than state-of-the-art methods.

1 Introduction

Reconstructing an image from (noisy) linear observations is maybe the most relevant inverse problem for modern image processing and appears in various applications like medical imaging and astronomy [7]. If the latent image is n -dimensional, for n ∈ N , it is well-known that Ω( n ) observations are required for robust identification in general. In practice, imaging problems are however often illposed, i.e., the number of observations is smaller than n or the operator creating the observations is defective [88, 93]. In such situations, the fundamental lower bound of Ω( n ) can be relaxed by leveraging structural priors of the latent image in the reconstruction process.

Of the various priors that are used for solving ill-posed inverse problems in the literature, sparsity 1 and low-rankness are most prevalent. This prominent role can be explained with their competitive performance in imaging tasks and the rigorous mathematical analysis they allow [10, 68]. For instance, consider the recovery of an n 1 × n 2 -dimensional image X ⋆ ∈ R n 1 × n 2 from linear observations

where A : R n 1 × n 2 → R m is a linear operator modeling the impulse response of the sensing device and η ∈ R m models additive noise. Whereas this problem is ill-posed for m < n 1 n 2 , it has been established [38, 15, 78] that it becomes well-posed if X ⋆ is sparse or of low rank. The aforementioned works prove that m = Ω( s 1 s 2 ) observations suffice for robust reconstruction if X ⋆ is s 1 -row-sparse

and s 2 -column-sparse, and that m = Ω( r ( n 1 + n 2 )) observations suffice if X ⋆ is a rankr matrix. These bounds, which relax the general lower bound of m = Ω( n 1 n 2 ) , agree with the degrees of freedom of sparse and low-rank matrices, respectively.

A number of computationally challenging problems in signal processing and machine learning can be formulated as instances of (1) with X ⋆ being simultaneously structured , i.e., X ⋆ is both of rank r and s 1 -row-sparse/ s 2 -column-sparse. Examples encompass sparse phase retrieval [51, 46, 11, 47, 85], sparse blind deconvolution [59, 83], hyperspectral imaging [43, 41, 89, 102], sparse reduced-rank regression [22, 100], and graph denoising and refinement [80, 103]. In these settings, the hope is that due to leveraging the simultaneous structure, Ω( r ( s 1 + s 2 )) observations suffice to identify the data matrix. For r ≪ s 1 , s 2 ≪ n 1 , n 2 , these bounds are significantly smaller than the bounds for single-structured data matrices.

From an algorithmic point of view, however, the simultaneously structured recovery problem poses obstacles that are not present for problems where X ⋆ is only of low-rank or (group) sparse: In the latter case, variational methods [6] that formulate the reconstruction method in terms of optimizing a suitable objective with a structural regularization term involving ℓ p / ℓ 2 ,p -(quasi-)norms and and S p -Schatten (quasi-)norms have been well-understood, leading to tractable algorithms in the information theoretically optimal regime [28, 17, 38].

For simultaneously structured problems, on the other hand, Oymak et al. showed in [74] that a mere linear combination of convex regularizers for different sparsity structures - in our case, nuclear and ℓ 2 , 1 -norms - cannot outperform recovery guarantees of the 'best' one of them alone. While this indicates that leveraging two priors at once is a way more intricate problem than leveraging a single prior, it was also shown in [74] that minimizing a linear combination of rank and row sparsity can indeed lead to guaranteed recovery from Ω( r ( s 1 + s 2 )) measurements. The downside is that the combination of these non-convex and discontinuous quantities does not lend itself directly to practical optimization algorithms, and to the best of our knowledge, so far, there have been no works directly tackling the optimization of a combination of non-convex surrogates that come with any sort of convergence guarantees.

1.1 Contribution

In this work, we approach the reconstruction of simultaneously sparse and low-rank matrices by leveraging the positive results of [74] for non-convex regularizers. To this end, we introduce a family of non-convex, but continuously differentiable regularizers that are tailored to the recovery problem for simultaneously structured data. The resulting objectives lend themselves to efficient optimization by a novel algorithm from the class of iteratively reweighted least squares ( IRLS ) [26, 35, 70, 5, 1, 54, 55], the convergence of which we analyze in the information theoretically (near-)optimal regime. Specifically, our main contributions are threefold:

(i) In Algorithm 1, we propose a novel IRLS method that is tailored to leveraging both structures of the latent solution, sparsity and low-rankness, at once. The core components of the algorithm are the weight operator defined in Definition 2.1 and the update of the smoothing parameters in (12). Notably, the algorithm automatically balances between its low-rank and its sparsity promoting terms, leading to a reliable identification of s 1 -row-sparse and rankr ground truths 2 .

(ii) Under the assumption that A behaves almost isometrically on the set of row-sparse and low-rank matrices, we show in Theorem 2.5 that locally Algorithm 1 exhibits quadratic convergence towards X ⋆ . Note that if A is, e.g., a Gaussian operator, the isometry assumption is (up to log factors) fulfilled in the information theoretic (near-)optimal regime m = Ω( r ( s 1 + n 2 )) [60].

(iii) Finally, in Section 2.3 we identify the underlying family of objectives that are minimized by Algorithm 1. To make this precise, we define for τ > 0 and e denoting Euler's number the real-valued function f τ : R → R such that

which is quadratic around the origin and otherwise logarithmic in its argument. Using this definition, we define for ε > 0 the ( ε -) smoothed log -determinant objective F lr ,ε : R n 1 × n 2 → R and for δ > 0 the ( δ -)smoothed sum of logarithmic row-wise ℓ 2 -norms objective F sp ,δ : R n 1 × n 2 → R such that

Combining the above, we further define the ( ε, δ -) smoothed logarithmic surrogate objective F ε,δ : R n 1 × n 2 → R as

In Theorem 2.6, we prove for any A that the iterates of Algorithm 1 minimize quadratic majorizations of F ε,δ and form a non-increasing sequence on F ε,δ . To the best of our knowledge, the proposed method is the so far only approach for recovering simultaneously sparse and low-rank matrices which combines local (quadratic) convergence with a rigorous variational interpretation.

The numerical simulations in Section 4 support our theoretical findings and provide empirical evidence for the efficacy of the proposed method.

1.2 Related Work

Sparse and Low-Rank Recovery. Whereas leveraging a single matrix structure like sparsity or low-rankness in the reconstruction process can easily be obtained by convex regularizers [78, 16], Oymak et al. [74] showed that, if one is interested in near-optimal sampling rates, one cannot expect comparably simple solutions for identifying simultaneously structured objects; a minimization of (4) with convex terms F lr ,ε ( · ) and F sp ,δ ( · ) would be only as good as using the one structure that is information theoretically more favorable. A closely related problem 3 that appears in statistical literature under the name Sparse Principal Component Analysis (SPCA) [104, 25] is known to be NP-hard in general [67]. Despite the the intrinsic hardness of simultaneously structured recovery problems, promising empirical results for hyperspectral image demixing were shown in [40], where minimization problems involving the sum of reweighted convex surrogates are solved by a proximal scheme based on ADMM for the case of simultaneously sparse, low-rank and non-negative matrices. For the problem of simultaneously sparse and low-rank matrix recovery, there exist only a handful approaches that come with rigorous theoretical analysis. The first line of works [4, 37] aims to overcome the aforementioned limitations of purely convex methods in a neat way. They assume that the operator A has a nested structure such that basic solvers for low-rank resp. row/column-sparse recovery can be applied in two consecutive steps. Despite being an elegant idea, this approach clearly restricts possible choices for A and is of hardly any practical use.

In a second line of work, Lee et al. [60] consider general impulse response operators that satisfy a suitable restricted isometry property for s 1 -row- and s 2 -column-sparse rankr matrices. They propose and analyze a highly efficient, greedy method, the so-called Sparse Power Factorization (SPF) which is a modified version of power factorization [48] and uses hard thresholding pursuit [36] to enforce sparsity in addition. In particular, they show that if X ⋆ is rankR , has s 1 -sparse columns and s 2 -sparse rows, then m ≳ R ( s 1 + s 2 ) log(max { en 1 /s 1 , en 2 /s 2 } ) Gaussian observations suffice for robust recovery, which is up to the log-factor at the information theoretical limit we discussed above. The result however assumes a low noise level and requires that SPF is initialized by a, in general, intractable method. Only in the special case that X ⋆ is spiky, which means that the norms of non-zero rows/columns exhibit a fast decay, a tractable substitute for the initialization method works provably. The analysis of SPF has been extended to the blind deconvolution setup in [59]. In [100], a related approach that combines gradient descent of a smooth objective with hard thresholding is considered, for which the authors show linear convergence from a suitable intialization if the measurement operator satisfies a restricted strong convexity and smoothness assumption.

A third line of work, approaches the problem from a variational point of view. In [33, 69] the

authors aim at enhancing robustness of recovery by alternating minimization of an ℓ 1 -norm based multi-penalty functional. In essence, the theoretical results bound the reconstruction error of global minimizers of the proposed functional depending on the number of observations. Although the authors only provide local convergence guarantees for the proposed alternating methods, the theoretical error bounds for global minimizers hold for arbitrarily large noise magnitudes and a wider class of groundtruth matrices than the one considered in [60].

The works [37, 29], which build upon generalized projection operators to modify iterative hard thresholding to the simultaneous setting, share the lack of global convergence guarantees.

In [79], the authors examine the use of atomic norms to perform recovery of simultaneously sparse and low-rank matrices, which uses a related, but different sparsity assumption compared to the row or column sparsity studied here. From a practical point of view, the such norms are hard to compute and the paper only proposes a heuristic polynomial time algorithm for the problem.

Finally, the alternative approach of using optimally weighted sums or maxima of convex regularizers [52] requires optimal tuning of the parameters under knowledge of the ground-truth.

Iteratively Reweighted Least Squares. The herein proposed iteratively reweighted least squares algorithm builds on a long line of research on IRLS going back to Weiszfeld's algorithm proposed in the 1930s for a facility location problem [95, 5]. IRLS is a practical framework for the optimization of non-smooth, possibly non-convex, high-dimensional objectives that minimizes quadratic models which majorize these objectives. Due to its ease of implementation and favorable data-efficiency, it has been widely used in compressed sensing [42, 18, 26, 57, 34, 55], robust statistics [45, 2, 72], computer vision [19, 61, 84], low-rank matrix recovery and completion [35, 70, 56, 54], and in inverse problems involving group sparsity [21, 101, 20]. Recently, it has been shown [62] that dictionary learning techniques can be incorporated into IRLS schemes for sparse and low-rank recovery to allow the learning of a sparsifying dictionary while recovering the solution. Whereas IRLS can be considered as a type of majorize-minimize algorithm [58], optimal performance is achieved if intertwined with a smoothing strategy for the original objective, in which case globally linear (for convex objectives) [26, 1, 72, 55, 75] and locally superlinear (for non-convex objectives) [26, 56, 54, 75] convergence rates have been shown under suitable conditions on the linear operator A .

However, there has only been little work on IRLS optimizing a sum of heterogenous objectives [81] -including the combination of low-rank promoting and sparsity-promoting objectives - nor on the convergence analysis of any such methods. The sole algorithmically related approach for our setting has been studied in [23], where a method has been derived in a sparse Bayesian learning framework, the main step of which amounts to the minimization of weighted least squares problems. Whereas the algorithm of [23] showcases that such a method can empiricially identify simultaneously structured matrices from a small number of measurements, no convergence guarantees or rates have been provided in the information-theoretically optimal regime. Furthermore, [23] only focuses on general sparsity rather than row or column sparsity.

1.3 Notation

We denote matrices and vectors by bold upper- and lower-case letters to distinguish them from scalars and functions. We furthermore denote the i -th row of a matrix Z ∈ R n 1 × n 2 by Z i, : and the j -th column of Z by Z : ,j . We abbreviate n = min { n 1 , n 2 } . We denote the r -th singular value of a matrix Z ∈ R n 1 × n 2 by σ r ( Z ) . Likewise, we denote the in ℓ 2 -norm s -largest row of Z by ρ s ( Z ) . (To determine ρ s ( Z ) , we form the in ℓ 2 -norm non-increasing rearrangement of the rows of Z convention, sort rows with equal norm according to the row-index.) We use ◦ to denote the Hadamard (or Schur) product, i.e., the entry-wise product of two vectors/matrices.

and, by We denote the Euclidean ℓ 2 -norm of a vector z ∈ R n by ∥ z ∥ 2 . For Z ∈ R n 1 × n 2 , the matrix norms we use encompass the operator norm ∥ Z ∥ := sup ∥ w ∥ 2 =1 ∥ Zw ∥ 2 , the row-sum p -quasinorm ∥ Z ∥ p, 2 := ( ∑ n 1 i =1 ∥ Z i, : ∥ p 2 ) 1 /p , the row-max norm ∥ Z ∥ ∞ , 2 := max i ∈ [ n 1 ] ∥ Z i, : ∥ 2 , and the Schatten-p quasinorm ∥ Z ∥ S p := ( ∑ n r =1 σ r ( Z ) p ) 1 / p . Note that two special cases of Schatten quasinorms are the nuclear norm ∥ Z ∥ ∗ := ∥ Z ∥ S 1 and the Frobenius norm ∥ Z ∥ F := ∥ Z ∥ S 2 .

2 IRLS for Sparse and Low-Rank Reconstruction

Recall that we are interested in recovering a rankr and s -row-sparse matrix X ⋆ ∈ R n 1 × n 2 from m linear observations

i.e., A : R n 1 × n 2 → R m is linear. We write X ⋆ ∈ M n 1 ,n 2 r,s := M n 1 ,n 2 r ∩ N n 1 ,n 2 s , where M n 1 ,n 2 r ⊂ R n 1 × n 2 denotes the set of matrices with rank at most r and N n 1 ,n 2 s ⊂ R n 1 × n 2 denotes the set of matrices with at most s non-zero rows. For convenience, we suppress the indices n 1 and n 2 whenever the ambient dimension is clear from the context. In particular, we know that X ⋆ = U ⋆ Σ ⋆ V ∗ ⋆ , where U ⋆ ∈ N n 1 ,r s , Σ ⋆ ∈ R r × r , and V ⋆ ∈ R n 2 × r denote the reduced SVD of X ⋆ . Furthermore, the row supports of X ⋆ and U ⋆ (the index sets of non-zero rows of X ⋆ resp. U ⋆ ) are identical, i.e., supp( X ⋆ ) = supp( U ⋆ ) = S ⋆ ⊂ [ n 1 ] := { 1 , . . . , n 1 } .

2.1 How to Combine Sparse and Low-Rank Weighting

As discussed in Section 1, the challenge in designing a reconstruction method for (5) lies in simultaneously leveraging both structures of X ⋆ in order to achieve the optimal sample complexity of m ≈ r ( s + n 2 ) . To this end, we propose a novel IRLS -based approach in Algorithm 1. The key ingredient for computing the ( k +1) -st iterate X ( k +1) ∈ R n 1 × n 2 in Algorithm 1 is the multi-structural weight operator W X ( k ) ,ε k ,δ k : R n 1 × n 2 → R n 1 × n 2 of Definition 2.1, which depends on the current iterate X ( k ) .

denote the number of singular values σ ( k ) = ( σ ( k ) i ) r k i =1 of X ( k ) larger than ε k and furthermore be

the number of rows of X ( k ) with ℓ 2 -norm larger than δ k . Define the r k left and right singular vectors of X ( k ) as columns of U ∈ R n 1 × r k and V ∈ R n 2 × r k , respectively, corresponding to the leading singular values σ ( k ) = ( σ ( k ) i ) r k i =1 := ( σ i ( X ( k ) ) ) r k i =1 . We define the weight operator W X ( k ) ,ε k ,δ k : R n 1 × n 2 → R n 1 × n 2 at iteration k of Algorithm 1 as

where W lr X ( k ) ,ε k : R n 1 × n 2 → R n 1 × n 2 is its low-rank promoting part

with Σ ε k = max( σ ( k ) i /ε k , 1) and W sp X ( k ) ,δ k ∈ R n 1 × n 1 is its sparsity-promoting part, which is diagonal with

The matrices U ⊥ and V ⊥ are arbitrary complementary orthogonal bases for U and V that do do not need to be computed in Algorithm 1.

Algorithm 1 IRLS for simultaneously low-rank rand row-sparse matrices

1: Input: Linear operator A : R n 1 × n 2 → R m , data y ∈ R m , rank and sparsity estimates ˜ r and ˜ s .

3: for k = 1 to K do

2: Initialize k = 0 , W X (0) ,ε 0 ,δ 0 = Id and set δ k , ε k = ∞ .

4: Weighted Least Squares: Update iterate X ( k ) by

5: Update Smoothing : Compute ˜ r +1 -st singular value σ ( k ) ˜ r +1 := σ ˜ r +1 ( X ( k ) ) and ( ˜ s +1) -st largest row ℓ 2 -norm ρ ˜ s +1 ( X ( k ) ) of X ( k ) , update

6: Update Weight Operator: For r k and s k as in (6) and (7),

compute W X ( k ) ,ε k ,δ k in (8) via W lr X ( k ) ,ε k in (9) and W sp X ( k ) ,δ k in (10).

compute first r k singular triplets σ ( k ) ∈ R r k , U ∈ R n 1 × r k and V ∈ R n 2 × r k ,

7: end for

8: Output: X ( K ) .

Recall F ε k ,δ k , F lr ,ε k , and F sp ,δ k from (3)-(4). The high-level idea of Algorithm 1, as for other IRLS methods, is to minimize quadratic functionals, which we call Q lr ,ε k ( · | X ( k ) ) : R n 1 × n 2 → R and Q sp ,δ k ( · | X ( k ) ) : R n 1 × n 2 → R and define them by

that majorize F ε k ,δ k ( · ) (see Theorem 2.6 below) for any iteration k . This minimization leads to the weighted least squares problem (11) in Algorithm 1. This step can be implemented by standard numerical linear algebra (see the supplementary material for a discussion of its computational complexity). As a second ingredient of the method, the smoothing parameters ε k and δ k of F ε k ,δ k are updated (i.e., decreased) in step (12) before the weight operator is updated according to the current iterate information. For the weight operator update, it is only necessary to compute row norms and leading singular triplets of X ( k ) .

Remark 2.3. The particular form of the low-rank promoting part of the weight operator W lr X ( k ) ,ε k in (9) is due to [53, 54] and captures optimally spectral information both in the column and row space, unlike prior work on low-rank IRLS [35, 70], while retaining the property that the induced quadratic model Q lr ,ε k ( · | X ( k ) ) majorizes F lr ,ε ε k ( · ) (see proof of Theorem 2.6). This choice is critical to enable a fast local rate as established in Theorem 2.5.

2.2 Local Quadratic Convergence of IRLS

Our first main result states that Algorithm 1 exhibits quadratic convergence in a local neighborhood of X ⋆ , a property Algorithm 1 shares with several methods from the IRLS family. We only need to assume that A acts almost isometrically on the set M r,s .

Definition 2.4. We say that a linear operator A : R n 1 × n 2 → R m satisfies the rankr and rows -sparse restricted isometry property (or ( r, s ) -RIP) with RIP-constant δ ∈ (0 , 1) if

for all Z ∈ M r,s .

It is worth highlighting that Gaussian operators satisfy the above RIP with high-probability if m ≥ cr ( s + n 2 ) log( en 1 /s ) , for some absolute constant c > 0 , see for instance [60]. Up to logfactors, this is at the information theoretic limit which we discussed in the beginning. The convergence result for Algorithm 1 now reads as follows.

Theorem 2.5 (Local Quadratic Convergence) . Let X ⋆ ∈ M r,s be a fixed ground-truth matrix that is s -row-sparse and of rank r . Let linear observations y = A ( X ⋆ ) be given and assume that A has the ( r, s ) -RIP with δ ∈ (0 , 1) . Assume that the k -th iterate X ( k ) of Algorithm 1 with ˜ r = r and ˜ s = s updates the smoothing parameters in (12) such that one of the statements ε k = σ r +1 ( X ( k ) ) or δ k = ρ s +1 ( X ( k ) ) is true, and that r k ≥ r and s k ≥ s .

If X ( k ) satisfies

where c ∥A∥ 2 → 2 = √ 1 + ∥A∥ 2 2 → 2 (1 -δ ) and n = min { n 1 , n 2 } , then the local convergence rate is quadratic in the sense that

for

and X ( k + ℓ ) ℓ →∞ → X ⋆ .

The proof of Theorem 2.5 is presented in the supplementary material. To the best of our knowledge, so far no other method exists for recovering simultaneously sparse and low-rank matrices that exhibits local quadratic convergence. In particular, the state-of-the-art competitor methods [60, 37, 69, 29] reach a local linear error decay at best.

On the other hand, (14) is rather pessimistic since for Gaussian A the constant c ∥A∥ 2 → 2 scales like √ ( n 1 n 2 ) /m , which means that the right-hand side of (14) behaves like m 3 / 2 / ( n ( n 1 n 2 ) 3 / 2 ) , whereas we observe quadratic convergence in experiments within an empirically much larger convergence radius. Closing this gap between theory and practical performance is future work.

It is noteworthy that the theory in [60] - to our knowledge the only other related work explicitly characterizing the convergence radius - holds on a neighborhood of X ⋆ that is independent of the ambient dimension. The authors of [60] however assume that the RIP-constant decays with the conditioning number κ of A , a quantity that might be large in applications. Hence, ignoring log-factors the sufficient number of measurements in [60] scales like m = Ω( κ 2 r ( s 1 + n 2 )) . In contrast, Theorem 2.5 works for any RIP-constant less than one which means for m = Ω( r ( s 1 + n 2 )) .

2.3 IRLS as Quadratic Majorize-Minimize Algorithm

With Theorem 2.5, we have provided a local convergence theorem that quantifies the behavior of Algorithm 1 in a small neighbourhood of the simultaneously row-sparse and low-rank ground-truth X ⋆ ∈ R n 1 × n 2 . The result is based on sufficient regularity of the measurement operator A , which in turn is satisfied with high probability if A consists of sufficiently generic random linear observations that concentrate around their mean.

In this section we establish that, for any measurement operator A , Algorithm 1 can be interpreted within the framework of iteratively reweighted least squares ( IRLS ) algorithms [26, 70, 75], which implies a strong connection to the minimization of a suitable smoothened objective function. In our case, the objective F ε,δ in (4) is a linear combination of sum-of-logarithms terms penalizing both non-zero singular values [31, 70, 14] as well as non-zero rows of a matrix X [50].

We show in Theorem 2.6 below that the IRLS algorithm Algorithm 1 studied in this paper is based on minimizing at each iteration quadratic models that majorize F ε,δ , and furthermore, that the iterates ( X ( k ) ) k ≥ 1 of Algorithm 1 define a non-increasing sequence ( F ε k ,δ k ( X ( k ) ) ) k ≥ 1 with respect to the objective F ε,δ of (4). The proof combines the fact that for fixed smoothing parameters ε k and δ k , the weighted least squares and weight update steps Algorithm 1 can be interpreted as a step of a Majorize-Minimize algorithm [86, 58], with a decrease in the underlying objective (4) for updated smoothing parameters.

Theorem 2.6. Let y ∈ R m , let the linear operator A : R n 1 × n 2 → R m be arbitrary. If ( X ( k ) ) k ≥ 1 is a sequence of iterates of Algorithm 1 and ( δ k ) k ≥ 1 and ( ε k ) k ≥ 1 are the sequences of smoothing parameters as defined therein, then the following statements hold.

The quadratic model functions Q lr ,ε k ( · | X ( k ) ) and Q sp ,δ k ( · | X ( k ) ) defined in (13) globally majorize the ( ε k , δ k ) -smoothed logarithmic surrogate objective F ε k ,δ k , i.e., for any Z ∈ R n 1 × n 2 , it holds that

The sequence ( F ε k ,δ k ( X ( k ) ) ) k ≥ 1 is non-increasing.

If ε := lim k →∞ ε k > 0 and δ := lim k →∞ δ k > 0 , then lim k →∞ ∥ X ( k ) -X ( k +1) ∥ F = 0 . Furthermore, in this case, every accumulation point of ( X ( k ) ) k ≥ 1 is a stationary point of

The proof of Theorem 2.6 is presented in the supplementary material.

3 Discussion of Computational Complexity

It is well-known that the solution of the linearly constrained weighted least squares problem (11) can be written as

where W k -1 := W X ( k -1) ,ε k -1 ,δ k -1 is the weight operator (8) of iteration k -1 [26, 54]. In [54, Theorem 3.1 and Supplementary Material], it was shown that in the case of low-rank matrix completion without the presence of a row-sparsity inducing term, this weighted least squares problem can be solved by solving an equivalent, well-conditioned linear system via an iterative solver that uses the application of a system matrix whose matrix-vector products have time complexity of O ( mr + r 2 max( n 1 , n 2 )) .

In the case of Algorithm 1, the situations is slightly more involved as we cannot provide an explicit formula for the inverse of the weight operator W k -1 as it amounts to the sum of the weight operators W lr X ( k -1) ,ε k -1 and W sp X ( k -1) ,δ k -1 that are diagonalized by different, mutually incompatible bases. However, computing this inverse is facilitated by the Sherman-Morrison-Woodbury formula [97]

for suitable matrices of compatible size E , F and invertible C , B and the fact that both W lr X ( k -1) ,ε k -1 and W sp X ( k -1) ,δ k -1 exhibit a 'low-rank plus (scaled) identity' or a 'sparse diagonal plus (scaled) identity' structure. After a simple application of the SMW formula, (17) can be rewritten such that the computational bottleneck becomes the assembly and inversion of a O ( r k max( n 1 , n 2 )) linear system. We note that in general, this can be done exactly in a time complexity of O ( r 3 k max ( n 1 , n 2 ) 3 ) using standard linear algebra. A crucial factor in the computational cost of the method is also the structure of the measurement operator A defining the problem, as the application of itself and its adjoint can significantly influence the per-iteration cost of IRLS; for dense Gaussian measurements, just processing the information of A amounts to mn 1 n 2 flops. If rank-one or Fourier-type measurements are taken, this cost can significantly be reduced, see [29, Table 1 and Section 3] for an analogous discussion.

We refer to the MATLAB implementation available in the repository https://github.com/ ckuemmerle/simirls for further details. While our implementation is not optimized for large-scale problems, the computational cost of Algorithm 1 was observed to be comparable to the implementations of SPF or RiemAdaIHT provided by the authors [69, 29]. We leave further improvements and adaptations to large-scale settings to future work.

4 Numerical Evaluation

In this section, we explore the empirical performance of IRLS in view of the theoretical results of Theorems 2.5 and 2.6, and compare its ability to recover simultaneous low-rank and rowsparse data matrices with the state-of-the-art methods Sparse Power Factorization (SPF) [60] and Riemannian adaptive iterative hard thresholding (RiemAdaIHT) [29], which are

among the methods with the best empirical performance reported in the literature. The method ATLAS [33] and its successor [69] are not used in our empirical studies since they are tailored to robust recovery and yield suboptimal performance when seeking high-precision reconstruction in low noise scenarios. We use spectral initialization for SPF and RiemAdaIHT . The weight operator of IRLS is initialized by the identity as described in Algorithm 1, solving an unweighted least squares problem in the first iteration. A detailed description of the experimental setup can be found in Appendix A.1.

Performance in Low-Measurement Regime. Figures 1 and 2 show the empirical probability of successful recovery when recovering s -row sparse ground-truths X ⋆ ∈ R 256 × 40 of rank r = 1 (resp. r = 5 ) from Gaussian measurements under oracle knowledge on r and s . The results are averaged over 64 random trials. As both figures illustrate, the region of success of IRLS comes closest to the information theoretic limit of r ( s + n 2 -r ) which is highlighted by a red line, requiring a significantly lower oversampling factor than the baseline methods.

10

20

30

40

50

100

150

200

Row sparsity

s

Nr. of measurements

m

10

20

30

40

50

100

150

200

10

20

30

40

50

100

150

200

0

0

.

2

0

.

4

0

.

6

0

.

8

1

20

40

60

400

600

800

1

,

000

Number of measurements

m

20

40

60

400

600

800

1

,

000

20

40

60

400

600

800

1

,

000

20

40

60

400

600

800

1

,

000

Row sparsity

s

20

40

60

400

600

800

1

,

000

20

40

60

400

600

800

1

,

000

In Appendix A.2 and Appendix A.3 in the supplementary material, we report on similar experiments conducted for other measurement operators than dense Gaussians, in which cases the empirical relative behavior of the methods is comparable.

Sensitivity to Parameter Choice. In applications of our setting, the quantities r and s might be unknown or difficult to estimate. In the second row of Figure 2, we repeat the experiment of the first row (rank5 ground truth), but run the algorithms with rank and sparsity estimates of ˜ r = 2 r and ˜ s = ⌊ 1 . 5 s ⌋ . Whereas all considered methods suffer a deterioration of performance, we observe that IRLS deteriorates relatively the least by a large margin. Furthermore, we observe that even if IRLS does not recovery X ⋆ , it converges typically to a matrix that is still low-rank and row-sparse (with

larger r and s ) satisfying the data constraint, while the other methods fail to convergence to such a matrix.

Convergence Behavior. Finally, we examine the convergence rate of the iterates to validate the theoretical prediction of Theorem 2.5 in the setting of Figure 2. Figure 3 depicts in log-scale the approximation error over the iterates of SPF , RiemAdaIHT , and IRLS . We observe that the IRLS indeed exhibits empirical quadratic convergence within a few iterations (around 10 ), whereas the other methods clearly only exhibit linear convergence. The experiment further suggests that the rather pessimistic size of the convergence radius established by Theorem 2.5 could possibly be improved by future investigations.

0

7

14

21

28

35

42

49

56

63

70

77

84

91

98

105 112

10

-

11

10

-

6

10

-

1

Iteration

k

Rel. Frobenius error

IRLS-LowRankRowSparse

Sparse Power Factorization [60]

Further experiments. In Appendix A.4, we provide additional experiments investigating the selfbalancing property of the objective (4), as well as the experiments on the noise robustness of the method in Appendix A.5.

5 Conclusion, Limitations and Future Work

Conclusion. In this paper, we adapted the IRLS framework to the problem of recovering simultaneously structured matrices from linear observations focusing on the special case of row sparsity and low-rankness. Our convergence guarantee Theorem 2.5 is hereby the first one for any method minimizing combinations of structural surrogate objectives that holds in the information-theoretic near-optimal regime and exhibits local quadratic convergence. The numerical experiments we conducted for synthetic data suggest that, due to its weak dependence on the choice of hyperparameters, IRLS in the form of Algorithm 1 can be a practical method for identifying simultaneously structured data even in difficult problem instances.

Limitations and Future Work. As in the case of established IRLS methods that optimize nonconvex surrogate objectives representing a single structure [26, 56, 54, 75], the radius of guaranteed quadratic convergence in Theorem 2.5 is the most restrictive assumption. Beyond the interpretation in terms of surrogate minimization as presented in Theorem 2.6, which holds without any assumptions on the initialization, our method shares a lack of a global convergence guarantees with other non-convex IRLS algorithms [70, 54, 75].

The generalization and application of our framework to combinations of structures beyond rank and row- (or column-)sparsity lies outside the scope of the present paper, but could involve subspacestructured low-rankness [31, 24, 99, 87] or analysis sparsity [30, 77]. A generalization of the presented IRLS framework to higher-order objects such as low-rank tensors is of future interest as convexifications of structure-promoting objectives face similar challenges [71, 74, 98] in this case.

In parameter-efficient deep learning, both sparse [39, 44, 32, 91] and low-rank [96, 94, 82] weight parameter models have gained considerable attention due to the challenges of training and storing, e.g., in large transformer-based models [92, 9]. It will be of interest to study whether in this non-linear case IRLS -like preconditioning of the parameter space can find network weights that are simultaneously sparse and low-rank, and could potentially lead to further increases in efficiency.

References

Deeksha Adil, Richard Peng, and Sushant Sachdeva. Fast, provably convergent IRLS algorithm for p-norm linear regression. Advances in Neural Information Processing Systems (NeurIPS 2019) , 2019.

Khurrum Aftab and Richard Hartley. Convergence of iteratively re-weighted least squares to robust m-estimators. In IEEE Winter Conference on Applications of Computer Vision (WACV 2015) , pages 480-487, 2015.

Ali Ahmed, Benjamin Recht, and Justin Romberg. Blind deconvolution using convex programming. IEEE Transactions on Information Theory , 60(3):1711-1732, 2014.

Sohail Bahmani and Justin Romberg. Near-optimal estimation of simultaneously sparse and low-rank matrices from nested linear measurements. Information and Inference: A Journal of the IMA , 5(3):331-351, 2016.

Amir Beck and Shoham Sabach. Weiszfeld's method: Old and new results. Journal of Optimization Theory and Applications , 164(1):1-40, 2015.

Martin Benning and Martin Burger. Modern regularization methods for inverse problems. Acta Numerica , 27:1-111, 2018.

Mario Bertero, Patrizia Boccacci, and Christine De Mol. Introduction to inverse problems in imaging . CRC Press, 2021.

Nicolas Boumal. An introduction to optimization on smooth manifolds . Cambridge University Press, 2023.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems (NeurIPS 2020) , 33:1877-1901, 2020.

Alfred M. Bruckstein, David L. Donoho, and Michael Elad. From sparse solutions of systems of equations to sparse modeling of signals and images. SIAM Review , 51(1):34-81, 2009.

T Tony Cai, Xiaodong Li, and Zongming Ma. Optimal rates of convergence for noisy sparse phase retrieval via thresholded wirtinger flow. The Annals of Statistics , pages 2221-2251, 2016.

T Tony Cai and Anru Zhang. Rate-optimal perturbation bounds for singular subspaces with applications to high-dimensional statistics. The Annals of Statistics , 46(1):60-89, 2018.

Patrizio Campisi and Karen Egiazarian. Blind Image Deconvolution: Theory and Applications . CRC Press, 2017.

E. J. Candès, Y. Eldar, T. Strohmer, and V. Voroninski. Phase Retrieval via Matrix Completion. SIAM J. Imag. Sci. , 6(1):199-225, 2013.

E. J. Candès and Y. Plan. Tight Oracle Inequalities for Low-Rank Matrix Recovery From a Minimal Number of Noisy Random Measurements. IEEE Trans. Inf. Theory , 57(4):2342-2359, 2011.

Venkat Chandrasekaran, Benjamin Recht, Pablo A Parrilo, and Alan S Willsky. The convex geometry of linear inverse problems. Foundations of Computational Mathematics , 12(6):805849, 2012.

Rick Chartrand. Exact reconstruction of sparse signals via nonconvex minimization. IEEE Signal Processing Letters , 14(10):707-710, 2007.

Rick Chartrand and Wotao Yin. Iteratively reweighted algorithms for compressive sensing. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 3869-3872, 2008.

Avishek Chatterjee and Venu Madhav Govindu. Robust relative rotation averaging. IEEE Transactions on Pattern Analysis and Machine Intelligence , 40(4):958-972, 2017.

Chen Chen, Lei He, Hongsheng Li, and Junzhou Huang. Fast iteratively reweighted least squares algorithms for analysis-based sparse reconstruction. Medical Image Analysis , 49:141152, 2018.

Chen Chen, Junzhou Huang, Lei He, and Hongsheng Li. Preconditioning for accelerated iteratively reweighted least squares in structured sparsity reconstruction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2014) , pages 2713-2720, 2014.

Lisha Chen and Jianhua Z Huang. Sparse reduced-rank regression for simultaneous dimension reduction and variable selection. Journal of the American Statistical Association , 107(500):1533-1545, 2012.

Wei Chen. Simultaneously sparse and low-rank matrix reconstruction via nonconvex and nonseparable regularization. IEEE Transactions on Signal Processing , 66(20):5313-5323, 2018.

Yuxin Chen and Yuejie Chi. Robust spectral compressed sensing via structured matrix completion. IEEE Transactions on Information Theory , 60(10):6576-6601, 2014.

Alexandre d'Aspremont, Laurent E Ghaoui, Michael I Jordan, and Gert R Lanckriet. A direct formulation for sparse PCA using semidefinite programming. In Advances in Neural Information Processing Systems (NIPS 2005) , pages 41-48, 2005.

I. Daubechies, R. DeVore, M. Fornasier, and C.S. Güntürk. Iteratively reweighted least squares minimization for sparse recovery. Commun. Pure Appl. Math. , 63:1-38, 2010.

Achiya Dax. On extremum properties of orthogonal quotients matrices. Linear Algebra and its Applications , 432(5):1234-1257, 2010.

David L. Donoho. Compressed sensing. IEEE Transactions on Information Theory , 52(4):12891306, 2006.

Henrik Eisenmann, Felix Krahmer, Max Pfeffer, and André Uschmajew. Riemannian thresholding methods for row-sparse and low-rank matrix recovery. Numerical Algorithms , pages 1-25, 2022.

Michael Elad and Michal Aharon. Image denoising via sparse and redundant representations over learned dictionaries. IEEE Transactions on Image Processing , 15(12):3736-3745, 2006.

Maryam Fazel, Haitham Hindi, and Stephen P. Boyd. Log-det heuristic for matrix rank minimization with applications to hankel and euclidean distance matrices. In Proceedings of the 2003 American Control Conference , volume 3, pages 2156-2162. IEEE, 2003.

William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. The Journal of Machine Learning Research , 23(1):5232-5270, 2022.

Massimo Fornasier, Johannes Maly, and Valeriya Naumova. Robust recovery of low-rank matrices with non-orthogonal sparse decomposition from incomplete measurements. Applied Mathematics and Computation , 392, 2021.

Massimo Fornasier, Steffen Peter, Holger Rauhut, and Stephan Worm. Conjugate gradient acceleration of iteratively re-weighted least squares methods. Comput. Optim. Appl. , 65(1):205259, 2016.

Massimo Fornasier, Holger Rauhut, and Rachel Ward. Low-rank matrix recovery via iteratively reweighted least squares minimization. SIAM Journal on Optimization , 21(4):1614-1640, 2011.

Simon Foucart. Hard thresholding pursuit: an algorithm for compressive sensing. SIAM Journal on Numerical Analysis , 49(6):2543-2563, 2011.

Simon Foucart, Rémi Gribonval, Laurent Jacques, and Holger Rauhut. Jointly low-rank and bisparse recovery: Questions and partial answers. Analysis and Applications , 18(01):25-48, 2020.

Simon Foucart and Holger Rauhut. A Mathematical Introduction to Compressive Sensing . Birkhäuser Basel, 2013.

Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations , 2019.

Paris V Giampouras, Konstantinos E Themelis, Athanasios A Rontogiannis, and Konstantinos D Koutroumbas. Simultaneously sparse and low-rank abundance matrix estimation for hyperspectral image unmixing. IEEE Transactions on Geoscience and Remote Sensing , 54(8):4775-4789, 2016.

Mohammad Golbabaee and Pierre Vandergheynst. Hyperspectral image compressed sensing via low-rank and joint-sparse matrix recovery. Proceedings of ICASSP 2012 , 2012.

Irina Gorodnitsky and Bhaskar Rao. Sparse signal reconstruction from limited data using focuss: A re-weighted minimum norm algorithm. IEEE Trans. Signal Process. , 45(3):600-616, 1997.

Benjamin D Haeffele and René Vidal. Structured low-rank matrix factorization: Global optimality, algorithms, and applications. IEEE Transactions on Pattern Analysis and Machine Intelligence , 42(6):1468-1482, 2019.

Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. The Journal of Machine Learning Research , 22(1):10882-11005, 2021.

Paul W. Holland and Roy E. Welsch. Robust regression using iteratively reweighted leastsquares. Communications in Statistics - Theory and Methods , 6(9):813-827, 1977.

Kishore Jaganathan, Samet Oymak, and Babak Hassibi. Sparse phase retrieval: Uniqueness guarantees and recovery algorithms. IEEE Transactions on Signal Processing , 65(9):24022410, 2017.

Gauri Jagatap and Chinmay Hegde. Fast, sample-efficient algorithms for structured phase retrieval. Advances in Neural Information Processing Systems (NeurIPS 2017) , 30, 2017.

Prateek Jain, Praneeth Netrapalli, and Sujay Sanghavi. Low-rank matrix completion using alternating minimization. Proceedings of the forty-fifth Annual ACM Symposium on Theory of Computing , pages 665-674, 2013.

Stuart M Jefferies and Julian C Christou. Restoration of astronomical images by iterative blind deconvolution. The Astrophysical Journal , 415:862, 1993.

Chengyu Ke, Miju Ahn, Sunyoung Shin, and Yifei Lou. Iteratively reweighted group lasso based on log-composite regularization. SIAM Journal on Scientific Computing , 43(5):S655S678, 2021.

Michael V. Klibanov, Paul E. Sacks, and Alexander V. Tikhonravov. The phase retrieval problem. Inverse problems , 11(1):1, 1995.

Martin Kliesch, Stanislaw J Szarek, and Peter Jung. Simultaneous structures in convex signal recovery-revisiting the convex combination of norms. Frontiers in Applied Mathematics and Statistics , 5:23, 2019.

Christian Kümmerle. Understanding and Enhancing Data Recovery Algorithms: From Noise-Blind Sparse Recovery to Reweighted Methods for Low-Rank Matrix Optimization . Dissertation, Technical University of Munich, 2019. Available at https://mediatum.ub. tum.de/doc/1521436/1521436.pdf .

Christian Kümmerle and Claudio Mayrink Verdun. A scalable second order method for ill-conditioned matrix completion from few samples. In International Conference on Machine Learning (ICML 2021) , 2021.

Christian Kümmerle, Claudio Mayrink Verdun, and Dominik Stöger. Iteratively reweighted least squares for basis pursuit with global linear convergence rate. Advances in Neural Information Processing Systems (NeurIPS 2021) , 2021.

Christian Kümmerle and Juliane Sigl. Harmonic mean iteratively reweighted least squares for low-rank matrix recovery. The Journal of Machine Learning Research , 19(1):1815-1863, 2018.

Ming-Jun Lai, Yangyang Xu, and Wotao Yin. Improved iteratively reweighted least squares for unconstrained smoothed ℓ q -minimization. SIAM Journal on Numerical Analysis , 51(2):927957, 2013.

Kenneth Lange. MMoptimization algorithms . SIAM, 2016.

Kiryung Lee, Yanjun Li, Marius Junge, and Yoram Bresler. Blind recovery of sparse signals from subsampled convolution. IEEE Transactions on Information Theory , 63(2):802-821, 2016.

Kiryung Lee, Yihong Wu, and Yoram Bresler. Near-optimal compressed sensing of a class of sparse low-rank matrices via sparse power factorization. IEEE Transactions on Information Theory , 64(3):1666-1698, 2018.

Seong Hun Lee and Javier Civera. Hara: A hierarchical approach for robust rotation averaging. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2022) , pages 15777-15786, 2022.

Stamatios Lefkimmiatis and Iaroslav Sergeevich Koshelev. Learning sparse and low-rank priors for image recovery via iterative reweighted least squares minimization. In The Eleventh International Conference on Learning Representations , 2023.

Adrian S. Lewis and Hristo S. Sendov. Nonsmooth Analysis of Singular Values. Part I: Theory. Set-Valued Analysis , 13(3):213-241, 2005.

Xiaodong Li, Shuyang Ling, Thomas Strohmer, and Ke Wei. Rapid, robust, and reliable blind deconvolution via nonconvex optimization. Applied and Computational Harmonic Analysis , 47(3):893-934, 2019.

He Lyu and Rongrong Wang. An exact sin θ formula for matrix perturbation analysis and its applications. arXiv preprint arXiv:2011.07669 , 2020.

C Ma, K Wang, Y Chi, and Yuxin Chen. Implicit regularization in nonconvex statistical estimation: Gradient descent converges linearly for phase retrieval, matrix completion, and blind deconvolution. Foundations of Computational Mathematics , 2020.

Malik Magdon-Ismail. Np-hardness and inapproximability of sparse pca. Information Processing Letters , 126:35-38, 2017.

Julien Mairal, Francis Bach, Jean Ponce, et al. Sparse modeling for image and vision processing. Foundations and Trends® in Computer Graphics and Vision , 8(2-3):85-283, 2014.

Johannes Maly. Robust sensing of low-rank matrices with non-orthogonal sparse decomposition. Applied and Computational Harmonic Analysis , 2023.

Karthik Mohan and Maryam Fazel. Iterative reweighted algorithms for matrix rank minimization. J. Mach. Learn. Res. , 13(1):3441-3473, 2012.

Cun Mu, Bo Huang, John Wright, and Donald Goldfarb. Square deal: Lower bounds and improved relaxations for tensor recovery. In Proceedings of the 31st International Conference on Machine Learning (ICML 2014) , volume 32 of Proceedings of Machine Learning Research , pages 73-81, 2014.

Bhaskar Mukhoty, Govind Gopakumar, Prateek Jain, and Purushottam Kar. Globallyconvergent iteratively reweighted least squares for robust regression problems. In International Conference on Artificial Intelligence and Statistics , pages 313-322, 2019.

Peter Ochs, Alexey Dosovitskiy, Thomas Brox, and Thomas Pock. On iteratively reweighted algorithms for nonsmooth nonconvex optimization in computer vision. SIAM Journal on Imaging Sciences , 8(1):331-372, 2015.

Samet Oymak, Amin Jalali, Maryam Fazel, Yonina C Eldar, and Babak Hassibi. Simultaneously structured models with application to sparse and low-rank matrices. IEEE Transactions on Information Theory , 61(5):2886-2908, 2015.

Liangzu Peng, Christian Kümmerle, and René Vidal. Global Linear and Local Superlinear Convergence of IRLS for Non-Smooth Robust Regression. In Advances in Neural Information Processing Systems (NeurIPS 2022) , volume 35, pages 28972-28987, 2022.

Liangzu Peng, Christian Kümmerle, and René Vidal. On the Convergence of IRLS and Its Variants in Outlier-Robust Estimation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2023) , 2023.

Saiprasad Ravishankar, Jong Chul Ye, and Jeffrey A Fessler. Image reconstruction: From sparsity to data-adaptive methods and machine learning. Proceedings of the IEEE , 108(1):86109, 2019.

Benjamin Recht, Maryam Fazel, and Pablo A. Parrilo. Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization. SIAM Review , 52(3):471-501, 2010.

Emile Richard, Guillaume R Obozinski, and Jean-Philippe Vert. Tight convex relaxations for sparse matrix factorization. Advances in Neural Information Processing Systems (NIPS 2014) , 27, 2014.

Emile Richard, Pierre-André Savalle, and Nicolas Vayatis. Estimation of simultaneously sparse and low rank matrices. In Proceedings of the 29th International Conference on Machine Learning (ICML 2012) , pages 51-58, 2012.

Masaki Samejima and Yasuyuki Matsushita. Fast general norm approximation via iteratively reweighted least squares. In Computer Vision-ACCV 2016 Workshops: ACCV 2016 International Workshops, Taipei, Taiwan, November 20-24, 2016 , pages 207-221. Springer, 2017.

Steffen Schotthöfer, Emanuele Zangrando, Jonas Kusch, Gianluca Ceruti, and Francesco Tudisco. Low-rank lottery tickets: finding efficient low-rank neural networks via matrix differential equations. In Advances in Neural Information Processing Systems (NeurIPS 2022) , volume 35, pages 20051-20063, 2022.

Laixi Shi and Yuejie Chi. Manifold gradient descent solves multi-channel sparse blind deconvolution provably and efficiently. IEEE Transactions on Information Theory , 67(7):47844811, 2021.

Yunpeng Shi, Cole M Wyeth, and Gilad Lerman. Robust group synchronization via quadratic programming. In International Conference on Machine Learning (ICML 2022) , pages 2009520105. PMLR, 2022.

Mahdi Soltanolkotabi. Structured signal recovery from quadratic measurements: Breaking sample complexity barriers via nonconvex optimization. IEEE Transactions on Information Theory , 65(4):2374-2400, 2019.

Ying Sun, Prabhu Babu, and Daniel P Palomar. Majorization-minimization algorithms in signal processing, communications, and machine learning. IEEE Transactions on Signal Processing , 65(3):794-816, 2016.

Yue Sun, Samet Oymak, and Maryam Fazel. Finite sample system identification: Optimal rates and the role of regularization. In Proceedings of the 2nd Conference on Learning for Dynamics and Control , volume 120, pages 16-25. PMLR, 10-11 Jun 2020.

Andrei Nikolajevits Tikhonov. Solution of incorrectly formulated problems and the regularization method. Soviet Math. , 4:1035-1038, 1963.

Christos G Tsinos, Athanasios A Rontogiannis, and Kostas Berberidis. Distributed blind hyperspectral unmixing via joint sparsity and low-rank constrained non-negative matrix factorization. IEEE Transactions on Computational Imaging , 3(2):160-174, 2017.

B. Vandereycken. Low-Rank Matrix Completion by Riemannian Optimization. SIAM J. Optim. , 23(2), 2013.

Mukund Varma T, Xuxi Chen, Zhenyu Zhang, Tianlong Chen, Subhashini Venugopalan, and Zhangyang Wang. Sparse winning tickets are data-efficient image recognizers. Advances in Neural Information Processing Systems (NeurIPS 2022) , 35:4652-4666, 2022.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems (NeurIPS 2017) , 30, 2017.

Curtis R Vogel. Computational methods for inverse problems . SIAM, 2002.

Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768 , 2020.

Endre Weiszfeld. Sur le point pour lequel la somme des distances de n points donnés est minimum. Tohoku Mathematical Journal, First Series , 43:355-386, 1937.

Wei Wen, Cong Xu, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Coordinating filters for faster deep neural networks. In Proceedings of the IEEE International Conference on Computer Vision (CVPR 2017) , pages 658-666, 2017.

Max A. Woodbury. Inverting modified matrices. Memorandum report , 42(106):336, 1950.

Quanming Yao, James Tin-Yau Kwok, and Bo Han. Efficient nonconvex regularized tensor completion with structure-aware proximal iterations. In International Conference on Machine Learning (ICML 2019) , pages 7035-7044. PMLR, 2019.

Jong Chul Ye, Yoseob Han, and Eunju Cha. Deep convolutional framelets: A general deep learning framework for inverse problems. SIAM Journal on Imaging Sciences , 11(2):991-1048, 2018.

Ming Yu, Varun Gupta, and Mladen Kolar. Recovery of simultaneous low rank and two-way sparse coefficient matrices, a nonconvex approach. Electronic Journal of Statistics , 14:413-457, 2020.

Zeinab Zeinalkhani and Amir H Banihashemi. Iterative reweighted ℓ 2 / ℓ 1 recovery algorithms for compressed sensing of block sparse signals. IEEE Transactions on Signal Processing , 63(17):4516-4531, 2015.

Yanbin Zhang, Long-Ting Huang, Yangqing Li, Kai Zhang, and Changchuan Yin. Low-rank and sparse matrix recovery for hyperspectral image reconstruction using bayesian learning. Sensors , 22(1):343, 2022.

Zhenyue Zhang, Zheng Zhai, and Limin Li. Graph refinement via simultaneously low-rank and sparse approximation. SIAM Journal on Scientific Computing , 44(3):A1525-A1553, 2022.

Hui Zou, Trevor Hastie, and Robert Tibshirani. Sparse principal component analysis. Journal of Computational and Graphical Statistics , 15(2):265-286, 2006.

Supplementary material for Recovering Simultaneously Structured Data via Non-Convex Iteratively Reweighted Least Squares

This supplement is structured as follows.

Appendix A presents some details about the experimental setup as well as additional numerical experiments.

Appendix B.1 presents the proof of Theorem 2.5.

Appendix B.2 presents the proof of Theorem 2.6.

Appendix C details some technical results that are used in Appendices B.1 and B.2.

A Experimental Setup and Supplementary Experiments

In this section, we elaborate on the detailed experimental setup that was used in Section 4 of the main paper. Furthermore, we provide additional experiments comparing the behavior of the three methods studied in Section 4 for linear measurement operators A that are closer to operators that can be encountered in applications of simultaneous low-rank and group-sparse recovery. Finally, we shed light on the evolution of the objective function (4) of IRLS (Algorithm 1), including in situations where the algorithm does not manage to recover the ground truth.

A.1 Experimental Setup

The experiments of Section 4 were conducted using MATLAB implementations of the three algorithms on different Linux machines using MATLAB versions R2019b or R2022b. In total, the preparation and execution of the experiments used approximately 1200 CPU hours. The CPU models used in the simulations are Dual 18-Core Intel Xeon Gold 6154, Dual 24-Core Intel Xeon Gold 6248R, Dual 8-Core Intel Xeon E5-2667, 28-Core Intel Xeon E5-2690 v3, 64-Core Intel Xeon Phi KNL 7210-F. For Sparse Power Factorization (SPF) [60], we used our custom implementation of [60, Algorithm 4 "rSPF_HTP"] and for Riemannian adaptive iterative hard thresholding (RiemAdaIHT) [29], we used an implementation provided to us by Max Pfeffer in private communications. We refer to Section 3 for implementation details for the IRLS method Algorithm 1.

In all phase transition experiments, we define successful recovery such that the relative Frobenius error ∥ X ( K ) -X ⋆ ∥ F ∥ X ⋆ ∥ F of the iterate X ( K ) returned by the algorithm relative to the simultaneously low-rank and row-sparse ground truth matrix X ⋆ is smaller than the threshold 10 -4 . As stopping criteria, we used the criterion that the relative change of Frobenius norm satisfies ∥ X ( k ) -X ( k -1) ∥ F ∥ X ( k ) ∥ F < tol for IRLS , the change in the matrix factors norms satisfy ∥ U k -U k -1 ∥ < tol and ∥ V k -V k -1 ∥ < tol for SPF , and the norm of the Riemannian gradient in RiemAdaIHT being smaller than tol for tol = 10 -10 , or if a maximal number of iterations is reached. This iteration threshold was chosen as max _ iter = 250 for IRLS and SPF and as max _ iter = 2000 for RiemAdaIHT , reflecting the fact that RiemAdaIHT is a gradient-type method which might need many iterations to reach a high-accuracy solution. The parameters were chosen so that the stopping criteria do not prevent a method's iterates reaching the recovery threshold if they were to reach X ⋆ eventually.

In the experiments, we chose random ground truths X ⋆ ∈ R n 1 × n 2 of rank r and row-sparsity s such that X ⋆ = ˜ X ⋆ / ∥ ∥ ∥ ˜ X ⋆ ∥ ∥ ∥ F , where ˜ X ⋆ = U ⋆ diag( d ⋆ ) V ∗ ⋆ , and where U ⋆ ∈ R n 1 × r is a matrix with s non-zero rows whose location is chosen uniformly at random and whose entries are drawn from i.i.d. standard Gaussian random variables, d has i.i.d. standard Gaussian entries and V ⋆ ∈ R n 2 × r has likewise i.i.d. standard Gaussian entries.

A.2 Random Rank-One Measurements

In Section 4, we considered only measurement operator A : R n 1 × n 2 → R m whose matrix representation consists of i.i.d. Gaussian entries, i.e., operators such that there are independent matrices

A 1 , . . . A m with i.i.d. standard Gaussian entries such that

for any X ∈ R n 1 × n 2 . While it is known that such Gaussian measurement operators satisfy the ( r, s ) -RIP of Section 4, which is the basis of our convergence theorem Theorem 2.5, in a regime of a near-optimal number of measurements with high probability, practically relevant measurement operators are often more structured; another downside of dense Gaussian measurements is that it is computationally expensive to implement their action on matrices.

In relevant applications of our setup, however, e.g., in sparse phase retrieval [46, 11, 47] or blind deconvolution [59, 83], the measurement operator consists of rank-one measurements. For this reason, we now conduct experiments in settings related to the ones depicted in Figure 1 and Figure 2 Section 4, but for random rank-one measurements where the action of A : R n 1 × n 2 → R m on X can be written as

for each j = 1 , . . . , m , where a j , b j are independent random standard Gaussian vectors. In Figure 4, we report the phase transition performance of RiemAdaIHT , SPF and IRLS for (256 × 40) -dimensional ground truths of different row-sparsities and different ranks if we are given such random rank-one measurements.

10

20

30

40

50

100

150

200

Number of measurements

m

10

20

30

40

50

100

150

200

10

20

30

40

50

100

150

200

20

40

60

400

600

800

1

,

000

Row sparsity

s

0

0

.

2

0

.

4

0

.

6

0

.

8

1

20

40

60

400

600

800

1

,

000

20

40

60

400

600

800

1

,

000

We observe in Figure 4 that compared to the setting of dense Gaussian measurements, the phase transitions of all three algorithms deteriorate slightly; especially for r = 1 , one can observe that the transition between no success and high empirical success rate is extends across a larger area. IRLS performs clearly best for both r = 1 and r = 5 , whereas SPF has the second best performance for r = 5 . For r = 1 , it is somewhat unclear whether RiemAdaIHT or SPF performs better.

A.3 Discrete Fourier Rank-One Measurements

We now revisit the experiments of Appendix A.2 for a third measurement setup motivated from blind deconvolution problems [3, 59, 64, 66, 83, 29], which are prevalent in astronomy, medical imaging and communications engineering [49, 13]. In particular, in these settings, if z ∈ R m is an (unknown) signal and w ∈ R m is an (unknown) convolution kernel, assume we are given the entries of their convolution ˜ y = z ∗ w . If we know that z = Au for some known matrix A ∈ R m × n 1 and an s -sparse vector u ∈ R n 1 and w = Bv for some known matrix B ∈ R m × n 2 and arbitrary vector v ∈ R n 2 , applying the discrete Fourier transform (represented via the DFT matrix F ∈ C m × m ), we can write the coordinates of

for each j = 1 , . . . , m , which allows us to write the problem as a simultaneously rank1 and s -row sparse recovery problem from Fourier-type measurements.

10

20

30

40

50

100

150

200

Number of measurements

m

10

20

30

40

50

100

150

200

10

20

30

40

50

100

150

200

20

40

60

400

600

800

1

,

000

Row sparsity

s

0

0

.

2

0

.

4

0

.

6

0

.

8

1

20

40

60

400

600

800

1

,

000

20

40

60

400

600

800

1

,

000

In Figure 5, we report the results of simulations with A and B chosen generically as standard real Gaussians for these Fourier-based rank1 measurements (including for rank5 ground truths, which goes beyond a blind deconvolution setting). We observe that the transition from no recovery to exact recovery for an increasing number of measurement (with fixed dimension parameters s , n 1 and n 2 ) happens earlier than for the random Gaussian rank-one measurements of Appendix A.2, but slightly later than for dense Gaussian measurements. Again, IRLS exhibits the best empirical data-efficiency with sharpest phase transition curves.

As a summary, we observe that IRLS is able to recovery simultaneously low-rank and row-sparse matrices empirically from fewer measurements than state-of-the-art methods for a variety of linear measurement operators, including in cases where the RIP assumption of Definition 2.4 is not satisfied and in cases that are relevant for applications.

A.4 Evolution of Objective Values

While Theorem 2.5 guarantees local convergence if the measurement operator A is generic enough and contains enough measurements (RIP-assumption), it is instructive to study the behavior of Algorithm 1 in situations where there are not enough measurements available to identify a specific low-rank and row-sparse ground truth X ⋆ which respect to which the measurements have been taken.

In this setting, Theorem 2.6 guarantees that the behavior of the IRLS methods is still benign as the sequence of ε - and δ -smoothed log-objectives ( F ε k ,δ k ( X ( k ) ) ) k ≥ 1 from (4) is non-increasing. In Figure 6, we illustrate the evolution of the relative Frobenius error of an iterate to the ground truth X ⋆ , the ( ε k , δ k ) -smoothed logarithmic surrogates F ε k ,δ k ( X ( k ) ) as well as of the rank and row-sparsity parts F lr ,ε k ( X ( k ) ) and F sp ,δ k ( X ( k ) ) of the objective, respectively, in two typical situations.

In particular, we can see the evolution of these four quantities in the setting of data of dimensionality n 1 = 128 , n 2 ∈ { 20 , 40 } , s = 20 and r = 5 created as in the other experiments, where a number of m = 875 and m = 175 (corresponding to an oversampling factor of 3 . 0 and 1 . 0 , respectively) dense Gaussian measurements are provided to Algorithm 1.

In the left plot of Figure 6, which corresponds to setting of abundant measurements, we observe that the four quantities all track each other relatively well on a semilogarithmic scale (note that we plot the square roots of the objective values to match the order of the (unsquared) relative Frobenius error),

converging to values between 10 -13 and 10 -11 (at which point the stopping criterion of the method applies) within 12 iterations.

In the second plot of Figure 6, the number of measurements exactly matches the number of degrees of freedom of the ground truth, in which case the X ( k ) does not converge to X ⋆ . However, it can be seen that Algorithm 1 still finds very meaningful solutions: It can be seen that within 86 iterations, F ε k ,δ k ( X ( k ) ) converges to ≈ 10 -12 (since √ F ε k ,δ k ( X ( k ) ) ≈ 10 -6 ) in a manner that is partially 'staircase-like': After 20 initial iterations where F ε k ,δ k ( X ( k ) ) decreases significantly at each iteration, its decrease is dominated by relatively sudden, alternating drops of the (blue) sparsity objective F sp ,δ k ( X ( k ) ) and the (red) rank objective F lr ,ε k ( X ( k ) ) , which typically do not occur simultaneously.

This illustrates the self-balancing property of the two objective terms in the IRLS objective F ε k ,δ k ( X ( k ) ) : while the final iterate at iteration k = 86 is not of the target row-sparsity s = 20 and r = 5 , it is still 20 -row sparse and has essentially rank 6 . This means that Algorithm 1 has found an alternative parsimonious solution to the simultaneous low-rank and row-sparse recovery problem that is just slightly less parsimonious.

Arguably, this robust performance in the low-data regime of IRLS is rather unique, and to the best of our knowledge, not shared by methods such as SPF or RiemAdaIHT , which typically breakdown in such a regime.

0

2

4

6

8

10

12

10

-

14

10

-

10

10

-

6

10

-

2

10

2

Rank Objective

√

F

lr,ε k

(

X

(

k

)

)

Sparsity Objective

√

F

sp,δ k

(

X

(

k

)

)

IRLS Objective

√

F

ε k ,δ k

(

X

(

k

)

)

Rel. Frob. error

∥

X

(

k

)

-

X

⋆

∥

F

/

∥

X

⋆

∥

F

0

20

40

60

80

10

-

7

10

-

6

10

-

5

10

-

4

10

-

3

10

-

2

10

-

1

10

0

iteration

k

iteration

k

A.5 Robustness under Noisy Measurements

The convergence theory for the IRLS method Algorithm 1 established in Theorem 2.5 assume that exact linear measurements y = A ( X ⋆ ) of a row-sparse and low-rank ground truth X ⋆ are provided to the algorithm. However, in practice, one would expect that the linear measurement model is only approximately accurate. For IRLS for sparse vector recovery, theoretical guarantees have been established for this case in [26, 57]. We do not extend such results to the simultaneously structured case, but we provide numerical evidence that IRLS as defined in Algorithm 1 can be used directly also for noisy measurements.

To this end, we conduct an experiment in the problem setup of Figure 1 in Section 4 for a fixed row-sparsity of s = 40 , in which the measurements provided to the algorithms IRLS , RiemAdaIHT

and SPF are such that

where w is a Gaussian vector (i.i.d. entries) with standard deviation of σ = √ ∥A ( X ⋆ ) ∥ 2 2 m · SNR and where SNR is a varying signal-to-noise ratio. We consider SNRs between 10 and 10 12 , and report the resulting relative Frobenius error statistics in Figure 7.

10

1

10

2

10

3

10

4

10

5

10

6

10

7

10

8

10

9

10

10

10

11

10

12

10

-

7

10

-

6

10

-

5

10

-

4

10

-

3

10

-

2

10

-

1

10

0

Signal-to-noise ratio SNR

relative Frobenius reconstruction error

∥

ˆ

X

-

X

0

∥

F

/

∥

X

0

∥

F

IRLS-LowRankRowSparse

Sparse Power Factorization [60]

We observe that the reconstruction error is consistently roughly proportional to the inverse square root of the signal-to-noise ratio, for all three algorithms considered. This suggests that IRLS is as noise robust as comparable algorithms, and expected to be return estimates of the original ground truth that has a reconstruction error that is of the order of the norm of the noise.

B Proofs

The following two sections contain the proofs of our main results. Let us begin with some helpful observations.

First note that the low-rank promoting part W lr X ( k ) ,ε k : R n 1 × n 2 → R n 1 × n 2 of our weight operator can be re-written as

where

Consequently, all weight operators in Definition 2.1 are self-adjoint and positive. Whereas for W sp X ( k ) ,δ k this is obvious, for W lr X ( k ) ,ε k it follows from the matrix representation

where D H ( σ ( k ) ,ε k ) ∈ R n 1 n 2 × n 1 n 2 is a diagonal matrix with the entries of H ( σ ( k ) , ε k ) , which are all positive, on its diagonal.

B.1 Proof of Theorem 2.5

Before approaching the proof of Theorem 2.5, let us collect various important observations. In order to keep the presentation concise, we defer part of the proofs to Appendix C.

For a rankr matrix Z = UΣV ∗ , we define the tangent space of the manifold of rankr matrices at Z as

In a similar manner, we can define for Z = UΣV ∗ ∈ M r,s and S = supp( Z ) = supp( U ) ⊂ [ n 1 ] the tangent space of M r restricted to S as

As the following lemma shows, orthogonal projections onto the sets M n 1 ,n 2 r , N n 1 ,n 2 s , T U , V , and T U , V ,S can be efficiently computed.

Lemma B.1. We denote the projection operators onto M n 1 ,n 2 r and N n 1 ,n 2 s by T r and H s . T r truncates a matrix to the r dominant singular values; H s sets all but the s in ℓ 2 -norm largest rows to zero. In case of ambiguities (multiple singular values/rows of same magnitude), by convention we choose the r (respectively s ) with smallest index.

For U and V fixed, the orthogonal projection onto T U , V is given by

For S ⊂ [ n 1 ] and U , V fixed with supp( U ) = S , the orthogonal projection onto T U , V ,S is given by

where P S projects to the row support S , i.e., it sets all rows to zero which are not indexed by S .

The proof of Lemma B.1 is provided in Appendix C.2. In contrast to the above named projections, the projection onto M n 1 ,n 2 r,s is not tractable. However, [29, Lemma 2.4] shows that locally P M r,s can be replaced by the concatenation of T r and H s , i.e., for Z ⋆ ∈ M r,s and Z ≈ Z ⋆ , one has that

For a matrix X ∈ R n 1 × n 2 and i ∈ [ n 1 ] , we set ρ i ( X ) = ∥ ( X ) i ′ , : ∥ 2 where i ′ is a row index corresponding to the i -th largest row of X in ℓ 2 -norm. More precisely, if X is a decreasing rearrangement of X with rows ordered by magnitude in ℓ 2 -norm, then ρ i ( X ) = ∥ ( X ) i, : ∥ 2 . As the following lemma shows, the quantity ρ s ( X ) determines a local neighborhood of X on which H s preserves the row-support.

Lemma B.2. Let X ∈ R n 1 × n 2 be a matrix with row-support S ⊂ [ n 1 ] and | S | = s . Then, for any Z ∈ R n 1 × n 2 with ∥ X -Z ∥ ∞ , 2 := max i ∈ [ n 1 ] ∥ X i, : -Z i, : ∥ 2 ≤ 1 2 ρ s ( X ) the matrix H s ( Z ) has row-support S .

Proof : Note that

implies that any non-zero row of X corresponds to a non-zero row of H s ( Z ) and hence yields the claim.

A first important observation is that if A has the ( r, s ) -RIP, then the norm of kernel elements of A is bounded in the following way.

LemmaB.3. If A has the ( r, s ) -RIP with δ ∈ (0 , 1) and U ∈ R n 1 × r , V ∈ R n 2 × r with supp( U ) = S , | S | ≤ s , then

for all Ξ ∈ ker( A ) .

The proof of Lemma B.3 is presented in Appendix C.3.

Remark B.4. If A is a Gaussian operator with standard deviation √ 1 m , one has with high probability that ∥A∥ 2 2 → 2 ≈ n 1 n 2 m .

We use of the following lemma to characterize the solution of the weighted least squares problem (11). Its proof is analogous to [54, Lemma B.7] and [26, Lemma 5.2].

Lemma B.5. Let A : R n 1 × n 2 → R m and y ∈ R m . Let W X ( k ) ,ε k ,δ k : R n 1 × n 2 → R n 1 × n 2 be the weight operator (8) defined based on the information of X ( k ) ∈ R n 1 × n 2 . Then the solution of the weighted least squares step (11) of Algorithm 1

is unique and solves (21) if and only if

For any iterate X ( k ) of Algorithm 1, we furthermore abbreviate the tangent space (20) of the fixed rankr manifold M r restricted to S at H s ( X ( k ) ) by

where ˜ U ∈ R n 1 × r and ˜ V ∈ R n 2 × r are matrices with leading 4 r singular vectors of H s ( X ( k ) ) as columns, and S ∈ [ n 1 ] is the support set of the s rows of X ( k ) with largest ℓ 2 -norm.

The following lemma is the first crucial tool for showing local quadratic convergence of Algorithm 1.

Lemma B.6. Let X ⋆ ∈ M r,s and let X ( k ) be the k -th iterate of Algorithm 1 with rank and sparsity parameters ˜ r = r and ˜ s = s , let δ k , ε k be such that s k and r k from Definition 2.1 satisfy s k ≥ s and r k ≥ r . Assume that there exists a constant c > 1 such that

where T k = T ˜ U , ˜ V ,S is as defined in (23) for matrices ˜ U ∈ R n 1 × r and ˜ V ∈ R n 2 × r of leading r left and right singular vectors of H s ( X ( k ) ) and S ⊂ [ n 1 ] is the support set of H s ( X ( k ) ) . Assume furthermore that

Then,

where ∥ M ∥ 1 , 2 = ∑ i ∥ M i, : ∥ 2 denotes the row-sum norm of a matrix M , and W lr X ( k ) ,ε k and W sp X ( k ) ,δ k are the weight operators (9) and (10) from Definition 2.1.

The proof of Lemma B.6 is presented in Appendix C.4.

Remark B.7. By revisiting the proof of Lemma B.6 (omit the bound in (48) and keep the term ⟨ Ξ , ¯ W Ξ ⟩ until the end), one can show under the same assumptions as in Lemma B.6 that

where U and V are containing the left and right singular vectors of X ( k ) , see Definition 2.1.

The contribution of the norms of the weighted X ⋆ terms in Lemma B.6 can be controlled by Lemmas B.8 and B.9 below.

Lemma B.8. Let W lr X ( k ) ,ε k : R n 1 × n 2 → R n 1 × n 2 be the rank-based weight operator (9) that uses the spectral information of X ( k ) and let X ⋆ ∈ R n 1 × n 2 be a rankr matrix. Assume that there exists 0 < ζ < 1 2 such that

Then for each 1 ≤ q ≤ ∞ ,

and

where K q is such that K q = 2 1 /q for 1 ≤ q ≤ 2 and 4 ≤ q , K q = 2 for 2 < q ≤ 4 and K q = 1 for q = ∞ .

Lemma B.9. Let W sp X ( k ) ,δ k ∈ R n 1 × n 1 be the row-sparsity-based weight operator (10) that uses the current iterate X ( k ) with δ k = min ( δ k -1 , ρ s +1 ( X ( k ) ) ) and let X ⋆ ∈ R n 1 × n 2 be an s -row-sparse matrix. Assume that there exists 0 < ζ < 1 2 such that

where ρ s ( M ) denotes the ℓ 2 -norm of the in ℓ 2 -norm s -largest row of M . Then

Lemma B.8 is a refined version of [54, Lemma B.9] the proof of which we omit here. 5 The proof of Lemma B.9 is provided in Appendix C.5. Finally, the following lemma will allow us to control the decay of the IRLS parameters δ k and ε k .

Lemma B.10 ([54, Lemma B.5]) . Let X ⋆ ∈ M r,s , assume that A has the ( r, s ) -RIP with δ ∈ (0 , 1) , and let us abbreviate n = min { n 1 , n 2 } .

Assume that the k -th iterate X ( k ) of Algorithm 1 with ˜ r = r and ˜ s = r updates the smoothing parameters in (12) such that one of the statements ε k = σ r +1 ( X ( k ) ) or δ k = ρ s +1 ( X ( k ) ) is true, and that r k ≥ r and s k ≥ s . Furthermore, let

Then

The proof of Lemma B.10 is provided in Appendix C.6. We finally have all the tools to prove Theorem 2.5. Note that (14) implies

and

Similarly, by assumption such that Lemma B.9 yields

which we will use in the proof below. The latter follows from the fact that for ˜ r = r

Proof of Theorem 2.5 : First note, that by assumption ˜ r = r and ˜ s = s . Furthermore, since 1 48 c 2 ∥A∥ 2 → 2 ≤ 1 2 , the closeness assumption (29) implies that H s ( X ( k ) ) and X ⋆ share the same support due to Lemma B.2. Let X ( k ) be the k -th iterate of Algorithm 1. Since the operator A : R n 1 × n 2 → R m has the ( r, s ) -RIP with δ ∈ (0 , 1) , Lemma B.3 yields for all U ∈ R n 1 × r , V ∈ R n 2 × r with supp( U ) = S , | S | ≤ s , that

for any Ξ ∈ ker( A ) . Furthermore, due to our assumption that ˜ s = s and ˜ r = r , the smoothing parameter update rules in (12), i.e., δ k = min ( δ k -1 , ρ s +1 ( X ( k ) ) ) and ε k = min ( ε k -1 , σ r +1 ( X ( k ) ) ) , imply that r k ≥ r and s k ≥ s for all k . We can thus apply Lemma B.6 for Ξ ( k ) := X ( k ) -X ⋆ (note at this point that (29) implies the closeness assumption (25) of Lemma B.6) and obtain

where W lr X ( k ) ,ε k : R n 1 × n 2 → R n 1 × n 2 is the low-rank promoting part (9) of the weight operator associated to X ( k ) and W sp X ( k ) ,δ k ∈ R n 1 × n 1 the sparsity promoting part (10). Since by assumption

Inserting (32) and (33) into (31) we obtain that

Due to the assertion that r k ≥ r , it holds that ε k ≤ σ r +1 ( X ( k ) ) . Therefore, Lemma C.3 yields that ( ε 2 k +2 ε k ∥ Ξ ( k ) ∥ +2 ∥ Ξ ( k ) ∥ 2 ) ≤ 5 ∥ Ξ ( k ) ∥ 2 .

and, since s k ≥ s , also that since δ k ≤ ρ s +1 ( X ( k ) ) in this case.

Thus, using the assertion that one of the statements ε k = σ r +1 ( X ( k ) ) or δ k = ρ s +1 ( X ( k ) ) is true, we obtain from (34) that

To show the remaining statement, we need to argue that the assertions of Theorem 2.5 are satisfied not only for k , but for any k + ℓ with ℓ ≥ 1 . For this, it is sufficient to show that

r k +1 ≥ r ,

s k +1 ≥ s ,

ε k +1 ≤ 1 48 σ r ( X ⋆ ) ,

(29) holds for X ( k +1) , and that

one of the statements ε k +1 = σ r +1 ( X ( k +1) ) or δ k +1 = ρ s +1 ( X ( k +1) ) is true,

as in this case, X ( k + ℓ ) ℓ →∞ → X ⋆ follows by induction due to successive application of (36).

For 1. and 2., we see that this follows from the smoothing parameter update rules (12) which imply that ε k +1 ≤ σ r +1 ( X ( k +1) ) and δ k +1 ≤ ρ s +1 ( X ( k +1) ) .

3. follows from (30) and the fact that due to (12), ( ε k ) k ≥ 1 is non-increasing. 4. is satisfied due to (36) and (29). To show 5., we note that due to (29), the assertion (28) is satisfied, and therefore it follows from (35) and Lemma B.10 that

We now distinguish the case (i) δ k < ε k and the case (ii) δ k ≥ ε k In case (i), it holds that

.

where the last inequality holds since by (29) the k -th iterate X ( k ) additionally satisfies

In this case, due to the smoothing parameter update rule (12), we have that ε k +1 = σ r +1 ( X ( k +1) ) . In case (ii), we have likewise that

due to (35), Lemma B.10, and (37). Hence, δ k +1 = ρ s +1 ( X ( k +1) ) which shows the remaining statement 5. and concludes the proof of Theorem 2.5.

B.2 Proof of Theorem 2.6

1.) Let ε, δ > 0 be arbitrary. Due to the additive structure of F ε,δ ( · ) , cf. (4), it is sufficient to establish that

for any Z , X ∈ R n 1 × n 2 , where W sp X ,δ : R n 1 × n 2 → R n 1 × n 2 is defined analogously to (10) and

for any Z , X ∈ R n 1 × n 2 , where W lr X ,ε : R n 1 × n 2 → R n 1 × n 2 is defined analogously to (9).

and the quadratic error decay

The argument for (38) is standard in the IRLS literature [2, 73, 76] and is based on the facts that both Q sp ,δ ( Z | X ) and F sp ,δ ( Z ) are row-wise separable, and that t ↦→ f √ τ ( √ t ) is concave and therefore majorized by its linearization: indeed, let g τ : R → R be such that

The function g τ ( · ) is continuously differentiable with derivative g ′ τ ( t ) = τ 2 max( | t | ,τ ) sign( t ) and furthermore, concave restricted to the non-negative domain R ≥ 0 .

Therefore, it holds for any t, t ′ ∈ R ≥ 0 that

We recall the definition f τ ( t ) = 1 2 τ 2 log( et 2 /τ 2 ) for | t | > τ and f τ ( t ) = 1 2 t 2 for | t | ≤ τ from (2) with derivative f ′ τ ( t ) = max( t 2 ,τ 2 ) t t 2 = τ 2 t max( t 2 ,τ 2 ) . Thus, for any x, z ∈ R , it follows that

and inserting τ = δ , z = ∥ Z i, : ∥ 2 , x = ∥ X i, : ∥ 2 and summing over i = 1 , . . . n 1 implies that

̸

From the chain rule, it follows that for all i = 1 , . . . , n 1 for which X i, : = 0 ,

and therefore

which shows the majorization of (38), recalling the definition W sp X ,δ = diag ( max ( ∥ X i, : ∥ 2 2 /δ 2 , 1) d 1 i =1 ) -1 ) of (10).

The majorization of (39) is non-trivial but follows in a straightforward way from [53, Theorem 2.4] as the objective F lr ,ε ( Z ) corresponds to the one of [53, Theorem 2.4] up to a multiplicative factor of ε 2 and constant additive factors, and since the weight operator W lr X ,ε corresponds to the weight operator used in [53, Chapter 2] for p = 0 .

2.) Due to the definition (3) of F sp ,δ k ( · ) and the derivative computation of (40), we observe that

comparing the resulting term with the definition of (10) of W sp X ( k ) ,δ k . Furthermore, an analogue equality follows from the the formula

with σ ( k ) i = σ i ( X ( k ) ) for any i ≤ d , which is a direct consequence from the calculus of spectral functions Lemma C.1, and inserting into the low-rank promoting weight operator formula (9)

Inserting ∇F sp ,δ k ( X ( k ) ) = W sp X ( k ) ,δ k · X ( k ) and ∇F lr ,ε k ( X ( k ) ) = W lr X ( k ) ,ε k ( X ( k ) ) into the definitions of Q lr ,ε k ( Z | X ( k ) ) and Q sp ,δ k ( Z | X ( k ) ) , we see that it holds that

and

Therefore, we see that the weighted least squares solution X ( k +1) of (11) for k +1 coincides with the minimizer of

with the weight operator W X ( k ) ,ε k ,δ k of (8), which implies that

Using the majorization (16) established in Statement 1 of Theorem 2.6 and (42), it follows that

using in the third line that Q lr ,ε k ( X ( k ) | X ( k ) ) = F lr ,ε k ( X ( k ) ) and Q sp ,δ k ( X ( k ) | X ( k ) ) = F sp ,δ k ( X ( k ) ) .

To conclude, it suffices to show that ε ↦→F ε,δ k ( X ( k +1) ) and δ ↦→F ε k ,δ ( X ( k +1) ) are non-decreasing functions, since (43) then extends to

where we used that the sequences ε k and δ k defined in Algorithm 1 are decreasing. So let us prove this last claim. We define for t ∈ R the function h t : R > 0 → R such that h t ( τ ) = f τ ( t ) , i.e.,

This function is continuously differentiable with h ′ t ( τ ) = 0 for all τ > | t | and

for τ < | t | , which implies that h ′ t ( τ ) ≥ 0 for all τ ≥ 0 and thus shows that ε ↦→F ε,δ k ( X ( k +1) ) and δ ↦→F ε k ,δ ( X ( k +1) ) are non-decreasing functions due to the additive structure of F ε,δ ( X ( k +1) ) and (4).

3.) First, we argue that ( X ( k ) ) k ≥ 1 is a bounded sequence: Indeed, if ε := lim k →∞ ε k > 0 and δ := lim k →∞ δ k > 0 , we note that

which implies that {∥ X ( k ) ∥} k ≥ 1 is bounded by a constant that depends on C X (1) .

Furthermore, we note that the optimality condition of (11) (see Lemma B.5) implies that X ( k +1) satisfies

Due to the definition of W ( k ) , we note that its smallest singular value (interpreted as matrix operator) can be lower bounded by

where c sp , X (1) and c lr , X (1) are constants that satisfy c sp , X (1) ≤ δ 2 exp( C X (1) /δ 2 -1) and c lr , X (1) ≤ ε 2 exp( C X (1) /ε 2 -1) .

Combining this with (44), the monotonicity according to Statement 2 of Theorem 2.6, and (41), it follows that

Summing over all k , this implies that lim k →∞ ∥ X ( k +1) -X ( k ) ∥ F = 0 .

Since ( X ( k ) ) k ≥ 1 is bounded, each subsequence of ( X ( k ) ) k ≥ 1 has a convergent subsequence. Let ( X ( k ℓ ) ) ℓ ≥ 1 be such a sequence with lim ℓ →∞ X ( k ℓ ) = ¯ X , i.e., ¯ X is an accumulation point of the sequence. As the weight operator W ( k ℓ ) depends continuously on X ( k ℓ ) , there exists a weight operator ¯ W : R n 1 × n 2 → R n 1 × n 2 such that ¯ W = lim ℓ →∞ W ( k ℓ ) .

Since lim k →∞ ∥ X ( k +1) -X ( k ) ∥ F = 0 , it also holds that X ( k ℓ +1) → ¯ X and therefore

for all Ξ ∈ ker A . The statement is shown as this is equivalent to ¯ X being a stationary point of F ε,δ ( · ) subject to the linear constraint { Z ∈ R n 1 × n 2 : A ( Z ) = y } .

C Technical addendum

C.1 Auxiliary Results

In the proof of Theorem 2.6, we use the following result about the calculus of spectral functions.

Lemma C.1 ([63],[35, Proposition 7.4]) . Let F : R d 1 × d 2 → R be a spectral function F = f ◦ σ with an associated function f : R d → R that is absolutely permutation symmetric. Then, F is differentiable at X ∈ R d 1 × d 2 if and only if f is differentiable at σ ( X ) ∈ R d .

In this case, the gradient ∇ F of F at X is given by

C.2 Proof of Lemma B.1

The projection operators for M n 1 ,n 2 r , N n 1 ,n 2 s , and T U , V are well-known, see e.g. [8]. To see the final statement assume that U has row-support S and note that P U , V ,S is idempotent, i.e.,

P U , V ,S P U , V ,S Z

-UU ∗ ( UU ∗ Z + P S ZVV ∗ -UU ∗ ZVV ∗ ) VV ∗

-UU ∗ ZVV ∗ -UU ∗ P S ZVV ∗ + UU ∗ ZVV ∗

One can easily check that P U , V ,S acts as identity when applied to matrices in T U , V ,S and that P U , V ,S = P ∗ U , V ,S since ⟨ Z ′ , P U , V ,S Z ⟩ F = ⟨ P U , V ,S Z ′ , Z ⟩ F , for any Z , Z ′ . This proves the claim.

C.3 Proof of Lemma B.3

C.4 Proof of Lemma B.6

In the proof of Lemma B.6 we will use the following fact.

Lemma C.2. Let W lr X ( k ) ,ε k be the weight operator defined in (9) , which is based on the matrices U ∈ R n 1 × r k and V ∈ R n 2 × r k of leading r k left and right singular vectors of X ( k ) . If M ∈ T U , V , then W lr X ( k ) ,ε k ( M ) ∈ T U , V . If M ∈ T ⊥ U , V , then W lr X ( k ) ,ε k ( M ) ∈ T ⊥ U , V .

Proof : If M ∈ T U , V , there exist M 1 ∈ R r k × r k , M 2 ∈ R r k × ( n 2 -r k ) , M 3 ∈ R ( n 1 -r k ) × r k such that

e.g., see [90, Proposition 2.1]. We thus observe that the weight operator W lr X ( k ) ,ε k : R n 1 × n 2 → R n 1 × n 2 from (18) satisfies

Similarly, if M ∈ T ⊥ U , V , there exists M 4 ∈ R ( n 1 -r k ) × ( n 2 -r k ) such that M = U ⊥ M 4 ( V ⊥ ) ∗ and lr

X

] (

H

]

[

0

0

(

k

σ

)

0

M

4

/ε

[

0

0

◦

, ε

k

)

][

2

k

V

∗

V

∗

⊥

0

M

4

∈

T

])[

⊥

U

,

V

.

V

V

∗

∗

⊥

]

W

(

k

)

,ε

k

(

M

) =

=

[

[

U

U

U

U

⊥

⊥

(

]

Proof of Lemma B.6 : Let Ξ ∈ R n 1 × n 2 be arbitrary. We start with some simple but technical observations. First note that by Lemma B.1, if T k = T ˜ U , ˜ V ,S ,

with

where we used that P ⊥ ˜ U , ˜ V Ξ = ( Id -P ˜ U , ˜ V ) Ξ = ˜ U ⊥ ˜ U ∗ ⊥ Ξ ˜ V ⊥ ˜ V ∗ ⊥ , for ˜ U ⊥ ∈ R n 1 × ( n 1 -r ) and ˜ V ⊥ ∈ R n 2 × ( n 2 -r ) being the complementary orthonormal bases of ˜ U and ˜ V , and that ˜ V ∗ ⊥ ˜ V = 0 .

Second, let now U ∈ R n 1 × r and V ∈ R n 2 × r be matrices with r leading left and right singular vectors of X ( k ) in their columns which coincide with the matrices U and V from Definition 2.1 in their first r columns. Then it follows from (45) and (46) that

By an argument analogous to (46), we observe that

where ˜ Ξ = P M ( Ξ ) is an element of the subspace M = M 1 ⊕M 2 ⊂ R n 1 × n 2 that is the direct sum of the subspaces M 1 := { P ⊥ U , V Z : Z ∈ R n 1 × n 2 } and M 2 := { P S c ZVV ∗ : Z ∈ R n 1 × n 2 } .

Let now W lr X ( k ) ,ε k be the rank promoting part of the weight operator from (9) and W sp X ( k ) ,δ k be the row-sparsity promoting part from (10). Note that the restriction of

to M is invertible as its first summand is invertible on M 1 = T ⊥ U , V , its second summand is invertible on M 2 (recall that the weight operators are positive definite), and M 1 ⊥ M 2 . Therefore it holds that

In the first inequality, we used that ¯ W |M is positive definite. In the second inequality, we used that σ min ( A + B ) ≥ σ min ( A ) + σ min ( B ) , for any positive semidefinite operators A and B , and and in the third inequality that ⟨ ˜ Ξ , ¯ W ˜ Ξ ⟩ ≤ ⟨ Ξ , ¯ W Ξ ⟩ . The latter observation can be deduced as follows: Note that, by the self-adjointness of ¯ W ,

Since ¯ W is positive semi-definite (due to the fact that both W lr X ( k ) ,ε k and W sp X ( k ) ,δ k are positive definite), all that remains is to argue that the mixed term on the right-hand side vanishes. To this end, note that P ⊥ U , V Z = P S c ZVV ∗ = 0 , for any Z ∈ M ⊥ and compute

We can now continue by estimating

using the positive semidefiniteness of W lr X ( k ) ,ε k and W sp X ( k ) ,δ k in the last inequality. To be precise, the last inequality can be argued as follows: Due to complimentary supports S and S c , we see that

Similarly, we note that W lr X ( k ) ,ε k acts diagonally on T U , V and T ⊥ U , V . Indeed, we have by Lemma C.2 that if M ∈ T U , V , then W lr X ( k ) ,ε k ( M ) ∈ T U , V and if M ∈ T ⊥ U , V , then W lr X ( k ) ,ε k ( M ) ∈ T ⊥ U , V , which implies

due to the orthogonality of elements in T ⊥ U , V and T U , V , respectively, and therefore it follows from Ξ = T U , V ( Ξ ) + T ⊥ U , V ( Ξ ) that

Combining the previous estimates with (47) and noticing that

we obtain

Next, we control the last two summands in (51) using matrix perturbation results. Recall that U ⋆ ∈ R n 1 × r and V ⋆ ∈ R n 2 × r are the singular vector matrices of the reduced singular value decomposition of X ⋆ . First observe that

Now note that, by [12, Lemma 1] and [65, Theorem 3.5], we obtain

̸

where we used some small observations in the last two inequalities: First, σ r +1 ( H s ( X ( k ) )) ≤ σ r +1 ( X ( k ) ) , which follows from the rectangular Cauchy interlacing theorem [27, Theorem 23]. Second, according to Lemma B.2 and (25), the row-support S of H s ( X ( k ) ) coincides with the row-support S ⋆ = { i ∈ [ n 1 ] : ∥ ( X ⋆ ) i, : ∥ 2 = 0 } of X ⋆ and hence H s ( X ( k ) ) -X ⋆ is a submatrix of X ( k ) -X ⋆ . Finally, σ r +1 ( X ( k ) ) = ∥ T r ( X ( k ) ) -X ( k ) ∥ ≤ ∥ X ⋆ -X ( k ) ∥ ≤ 1 48 σ r ( X ⋆ ) due to (25). Consequently,

and, by a similar argument,

such that it follows from P ⊥ U , V = Id -P U , V and P ⊥ ˜ U , ˜ V = Id -P ˜ U , ˜ V that

To estimate the fourth summand in (51), we argue analogously that

using again that σ r +1 ( X ( k ) ) ≤ ∥ X ⋆ -X ( k ) ∥ ≤ 1 48 σ r ( X ⋆ ) due to (25). Let now Ξ ( k +1) = X ( k +1) -X ⋆ . Combining (24) and (51)-(52) we can proceed to estimate that

where the third inequality follows from (8) and (25). Hence, rearranging (55) yields

By Lemma B.5, we know that X ( k +1) fulfills

which completes the proof. We used in the penultimate line Hölder's inequality and that

for all matrices A , B .

C.5 Proof of Lemma B.9

Note that by Lemma B.2 and (27), S := supp( H s ( X ( k ) )) = supp( X ⋆ ) . Since by assumption δ k ≤ ρ s ( X ( k ) ) we have by definition of W sp X ( k ) ,δ k that Z := W sp X ( k ) ,δ k · X ⋆ is a matrix with row-support S and rows

for i ∈ S . Now note that if (27) holds, then

where we used in the last estimate that with (27) and ∥ ( X ⋆ ) i, : ∥ 2 ≥ ρ s ( X ⋆ ) , for i ∈ S , we have

for all i ∈ S . The claim easily follows since Z has only s non-zero rows.

C.6 Proof of Lemma B.10

In the proof of Lemma B.10, we use a simple technical observation.

Lemma C.3. Let X ⋆ ∈ M r,s , let X ( k ) be the k -th iterate of Algorithm 1, and abbreviate Ξ ( k ) = X ( k ) -X ⋆ . Then the following two statements hold true:

Proof : By defining [ X ( k ) ] r to be the best rankr approximation of X ( k ) in any unitarily invariant norm, we bound

where the inequality follows the fact that X ⋆ is a rankr matrix.

Similarly, for the second statement, we have that

Proof of Lemma B.10 : First, we note that, using Lemma B.3, the observation in Remark B.7 yields

for all Ξ ∈ R n 1 × n 2 as the assumption (28) implies ∥ X ( k ) -X ⋆ ∥ ≤ min { 1 48 , 1 19 c } σ r ( X ⋆ ) . Thus, this holds in particular also for Ξ ( k ) = X ( k ) -X ⋆ . (Recall that U and V contain the leading singular vectors of X ( k ) , see Definition 2.1.) We estimate that

Furthermore, since max( ε k , ∥ Ξ ( k ) ∥ ) ≤ 1 48 σ r ( X ⋆ ) by assumption and ε k ≤ ∥ Ξ ( k ) ∥ by Lemma C.3, we can use a variant of Lemma B.8 to obtain

On the other hand, we note that Ξ ( k ) restricted to S c coincides with the restriction of X ( k ) to S c under assumption (28), cf. Lemma B.2, and therefore

With the estimate of above, this implies that

Inserting these estimates into (56), we obtain

If now either one of the two equations ε k = σ r +1 ( X ( k ) ) or δ k = ρ s +1 ( X ( k ) ) is true, it follows that

if the proximity condition ∥ Ξ ( k ) ∥ = ∥ X ( k ) -X ⋆ ∥ ≤ 1 21 c ∥A∥ 2 → 2 σ r ( X ⋆ ) is satisfied. Rearranging the latter inequality yields the conclusion of Lemma B.10.

Peer reviews:
comments: SUMMARY:
This paper proposes an IRLS method for recovering data with multiple, heterogeneous low-dimensional structures from linear observations. It combines non-convex surrogates for row-sparsity and rank, to identify simultaneously row-sparse and low-rank matrices from limited measurements. Theoretical results are provided that show locally quadratic convergence. The experiments demonstrate favorable empirical convergence and prove its efficacy in handling challenging data recovery scenarios.

STRENGTHS:
The paper is well-written overall and the contributions are clear. The challenge of the combination of structures is made apparent. The related work discussion is thorough. The IRLS is a popular approach and is practical even for large-scale systems. Experiments suggest the method is also robust to the choice of parameters r and s.

WEAKNESSES:
Although it is also true of other results in this area, only local convergence is guaranteed and practically it may be challenging to guarantee an initialization within the required radius. The results would be strengthened if other matrices besides Gaussian were shown to satisfy this RIP, and if others were even just used empirically and shown to still offer convergence.

QUESTIONS:
It would be good to remind the reader what \cal{F}_ functions are in (13).

Are there other matrices besides Gaussian that are known to satisfy the RIP given in Definition 2.3?

LIMITATIONS:
The authors discuss the non-optimality of the bounds in Theorem 2.4, arguing that experimentally the convergence radius appears to be much larger. The authors also discuss future work about generalizing to other combined structures, which the reviewer agrees is interesting future work but certainly not a limitation to the results in this paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-08

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis article introduces an algorithm for recovering jointly row-sparse and low-rank matrices from (underdetermined) linear measurements. The algorithm is based on iteratively reweighted least squares for a non-convex objective. The method is theoretically analysed, establishing local quadratic convergence rates under the restricted isometry property. The second theoretical result relates the algorithm more directly to IRLS to show that the objective is non-increasing and proves convergence to a stationary point.\n\nSTRENGTHS:\nI found the paper well-written, the problem of simultaneously recovering sparse and low rank matrices is challenging one cannot simply rely on standard convex regularisation. This work provides a nice solution to this problem via IRLS and the theoretical results are interesting: the authors establish both 'compressed sensing' type results that guarantee recovery under near optimal sampling complexity, as well as results that provide an understanding of the optimisation properties of the algorithm.\n\nWEAKNESSES:\n- The numerical results are somewhat limited, it would have been nice to have a discussion on the applications of the proposed method and more realistic numerical examples.\n- the result of Theorem 2.4 is restricted to the setting of exact measurements and does not cover robustness results, i.e. what if y has been corrupted with noise? Also, what if $X_*$ is only approximately sparse and low rank? Both of these are the more realistic settings, so it would have been nice to see a more complete result. Also, theory aside, equation (1) mentions additive noise, but it is not clear from the algorithm and results that the proposed method can handle additive noise.\n- Given that the title mentions \"simultaneously structured data\", I was expecting more than just low rank + sparse. It would be interesting to see the method extended to other kind of structures such as sparsity with respect to some dictionary.\n\nQUESTIONS:\n- can you mention the per-iteration complexity of your method and contrast it with existing methods?\n- the introduction mentions simultaneously low rank and both column and row sparse. Can your results be extended to this setting?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nPaper proposes to solve inverse problems on matrices subject to multiple types of sparsity (e.g. low-rank and element-wise sparsity) using an algorithm designed for the non-convex objective functions involved. The algorithm is a re-weighted least squares, which despite not solving a convex problem, authors prove a local convergence and a global consistency theorem for. They illustrate the performance of their algorithm on synthetic data.\n\nSTRENGTHS:\nThe algorithm proposed solves a difficult non-convex problem, and the theoretical results by authors are original and interesting.\n\nWEAKNESSES:\n1. The paper contains statistical results of the type \"if at least m measurements are available then the algorithm achieves good performance\", but is missing a more thorough discussion on algorithmic complexity. In particular, I would be curious to know if the truncated SVD in the Update Smoothing step is the heaviest computational piece of the algorithm or if the computational bottleneck is somewhere else. \n2. Numerical experiments are run on very small data. To really show the benefit of two sparsity types I would believe that bigger values of n1 n2 result in more convincing evidence that there is benefit in mixing the two prior knowledges\n\nQUESTIONS:\n1. what is the sensitivity of the algorithm to performing approximate SVDs\n2. Figure 3: fewer iterates are needed, OK, but what's the FLOP in each iterate? what's the overall computational cost of achieving epsilon close results as compared to others? is anything parallelizable?\n3. A related stream of research (see Tight Convex Relaxations for Sparse Matrix Factorization, NIPS 2014) characterizes the atoms that form the basis in which a doubly sparse matrix is sparse. How does this work, especially the lower bound Omega(r(s1+s2)), compare to those methods which directly minimize for the desired structure to recover?\n\nLIMITATIONS:\nbecause the computational complexity is not deeply discussed, it's hard to know how applicable the methods are to larger data", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies the problem of recovering a low-rank and row-sparse matrix from its compressed linear observations. A method based on iteratively re-weighted least squares is proposed, in which the sparsity inducing function is a non-convex log function. For theoretical contributions, this work provides a local quadratic convergence analysis of the iterates of the proposed algorithm under a restricted isometry property assumption, and it also studies the convergence of function value for any linear measurement operator using a Majorize-Minimize algorithm framework. Numerical experiments on synthetic datasets show that the proposed method can recover the ground truth with fewer measurements than some state-of-the-art methods.\n\nSTRENGTHS:\n1. This work provides convergence analyses not only for linear operators satisfying restricted isometry property but also for more general linear operators. Furthermore, it mentions when the restricted isometry property can be satisfied, and it also discusses the radius of the neighborhood in the local quadratic convergence.\n\n2. The proposed algorithm is well explained by identifying that the underlying family of objectives that are minimized during the iterations is (2).\n\n3. There are experiments for cases where the rank and sparsity are not accurately known as a prior, and thus the sensitivity to hyper-parameter choice is discussed, and also it provides a better simulation of potential real-world scenarios.\n\nWEAKNESSES:\n(1) The algorithm requires estimates of the rank and row sparsity, and the theoretical conclusions require that these estimates are accurate, i.e., $\\tilde r = r$ and $\\tilde s = s$. These priors and assumptions can limit the application of this method and its theoretical convergence guarantees.\n\n(2) There are some assumptions on the iterates $\\varepsilon_k$ and $\\delta_k$ in Theorem 2.4 and Theorem 2.5, but it is not clear whether these assumptions will ever be satisfied. Please refer to question (1) and (3) for details.\n\n(3) In numerical experiments, it would be better to report the time cost of different methods, including both the per iteration time cost and the total time cost.\n\nQUESTIONS:\n(1) In Theorem 2.4, will the assumptions $\\varepsilon_k = \\sigma_{r+1} (X^{(k)})$, $\\delta_k = \\rho_{s+1}(X^{k})$, and $\\varepsilon_k \\leq \\sigma_r(X_\\star) / 48$ ever be satisfied? \n\n(2) The inequality at the end of page 6 only holds for certain $k$ satisfying the assumptions, so how do we arrive at the conclusion $X^{(k+l)} \\to X_\\star$ from such an inequality for one step?\n\n(3) In Theorem 2.5, how strong/restricted is the assumption that $\\delta_k$ and $\\varepsilon_k$ have limits, and the limits are positive?\n\nLIMITATIONS:\nAs far as I see, there is no potential negative societal impact of this work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}