Iteratively Reweighted Least Squares for Basis Pursuit with Global Linear Convergence Rate

1 Introduction . The field of sparse recovery deals with the problem of recovering an ( approximately ) sparse vector x from only few linear measurements , presented by an underdetermined system of linear equations of the form y = Ax . One approach to solve this problem is to consider the ` 0-minimization under linear constraints , which is NP-hard in general [ 24 , 54 ] . For computational reasons , instead of ` 0-minimization , it is common practice to consider its convex relaxation min x∈RN ||x||1 subject to Ax = y , ( P1 ) ∗kuemmerle @ jhu.edu †verdun @ ma.tum.de ‡dominik.stoeger @ ku.de §The name order of the authors is alphabetical . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . where A ∈ Rm×N , b ∈ Rm are given , which is referred to as ` 1 ( -norm ) minimization [ 20 , 27 , 64 ] or basis pursuit [ 18 , 19 ] in the literature . Unlike ` 0-minimization , the optimization program ( P1 ) is computationally tractable in general , and a close relationship of their minimizers has been recognized and well-studied in the theory of compressive sensing [ 12 , 13 , 26 , 32 ] . In statistics and machine learning , an unconstrained variant of ( P1 ) , often called LASSO , amounts to the most well-studied tractable estimator for variable selection in high-dimensional inference [ 37 , 48 , 65 ] . ` 1-minimization has many other applications and it was even called the modern least squares [ 14 ] . The tractability of ( P1 ) becomes evident from the fact that it can be reformulated as a linear program [ 66 ] . However , as many problems of interest in applications are high-dimensional and therefore challenging for standard linear programming methods , many specialized solvers for ( P1 ) have been proposed , such as the Homotopy Method [ 28 ] , primal-dual methods [ 15 , 52 ] , Alternating Direction Method of Multipliers [ 9 ] , Bregman iterative regularization [ 71 ] and Semismooth Newton Augmented Lagragian Methods [ 46 ] and Iteratively Reweighted Least Squares ( IRLS ) , the latter of which is in the focus of this paper . Iteratively Reweighted Least Squares corresponds to a family of algorithms that minimizes nonsmooth objective functions by solving a sequence of quadratic problems , with its idea going back to a method proposed by Weiszfeld for the Fermat-Weber problem [ 7 , 69 ] . A variety of different problems such as robust regression in statistics [ 39 , 53 ] , total variation regularization in image processing [ 2 , 33 , 55 ] , joint learning of neural networks [ 72 ] , robust subspace recovery [ 45 ] and the recovery of low-rank matrices [ 31 , 42 , 43 , 51 ] can be solved efficiently by IRLS in practice , as it relies on simple linear algebra to solve the linear systems arising from the quadratic problems at each iteration , without the need of a careful initialization or intricate parameter tuning . On the other hand , the analysis of IRLS methods is typically challenging : General convergence results are often weak , and stronger convergence results are only available in particular cases ; see Section 2.3 for more details . IRLS for sparse recovery . In the sparse recovery context , the first variants of IRLS were introduced in [ 34 , 59 ] for the ` p-quasinorm minimization problem ( Pp ) with 0 < p ≤ 1 that is similar to ( P1 ) , but with ‖x‖p instead of ‖x‖1 as an objective . In [ 16 ] , modifications of the method of [ 34 , 59 ] using specific smoothing parameter update rules were observed to exhibit excellent numerical performance for solving ( Pp ) , retrieving the underlying sparse vector when most of the methods fail . A useful fact is that IRLS is one of the few methods ( ADMM being the other one [ 9 ] ) that provides a framework to solve both constrained and unconstrained formulations of ` p-minimization problems . A major step forward in the theoretical understanding of IRLS was achieved in [ 23 ] , where the authors showed that a variant of IRLS for ( P1 ) converges globally to an ` 1-minimizer if the measurement operator A fulfills the null space property of sufficient order , which essentially ensures that an ` 1-minimizer is actually sparse . However , since this proof relies on the existence of a convergent subsequence , their proof does not reveal any rate for global convergence . The analysis of [ 23 ] provides , furthermore , a locally linear convergence rate , but this local linear rate has the drawback that it only applies if the support of the true signal has been discovered , which is arguably the difficult part of ` 0-minimization—cf . Proposition 3.1 below and Section 4.1 . While several extensions and modifications of the IRLS algorithm in [ 23 ] have been proposed ( see , e.g. , [ 3 , 30 ] ) , this following fundamental algorithmic question has remained unanswered : What is the global convergence rate of the IRLS algorithm for ` 1-minimization ? Our contribution . We resolve this question , formally stated in [ 62 ] , and present a new IRLS algorithm that converges linearly to a sparse ground truth , starting from any initialization , as stated in Theorem 3.2 . Our algorithm returns a feasible solution with δ-accuracy , i.e. , ‖x∗ − xk‖1 ≤ δ , where x∗ is the underlying s-sparse vector , in k = O ( N √ ( logN ) /m log ( 1/δ ) ) iterations . Analogous to [ 23 ] , it is assumed that the measurement matrix A satisfies the so-called null space property [ 21 ] . We also provide a similar result for approximately sparse vectors . Our proof relies on a novel quantification of the descent of a carefully chosen objective function in the direction of the ground truth . Additionally , we support the theoretical claims by numerical simulations indicating that we capture the correct dimension dependence . We believe that the new analysis techniques in this paper are of independent interest and will pave the way for establishing global convergence rates for other variants of IRLS such as in low-rank matrix recovery [ 31 ] . Notation . We denote the cardinality of a set I by |I| and the support of a vector x ∈ RN , i.e. , the index set of its nonzero entries , by supp ( x ) = { j ∈ [ N ] : xj 6= 0 } . We call a vector s-sparse if at most s of its entries are nonzero . We denote by xI the restriction of x onto the coordinates indexed by I , and use the notation Ic : = [ N ] \ I to denote the complement of a set I . Furthermore , σs ( x ) ` 1 denotes the ` 1-error of the best s-term approximation of a vector x ∈ RN , i.e. , σs ( x ) ` 1 = inf { ‖x− z‖1 : z ∈ RN is s-sparse } . 2 IRLS for sparse recovery . We now present a simple derivation of the Iteratively Reweighted Least Squares ( IRLS ) algorithm for ` 1-minimization which is studied in this paper . IRLS algorithms can be interpreted as a variant of a Majorize-Minimize ( MM ) algorithm [ 63 ] , as we will lay out in the following . It mitigates the non-smoothness of the ‖ · ‖1-norm by using the smoothed objective function Jε : RN → R , which is defined , for a given ε > 0 , by Jε ( x ) : = N∑ i=1 jε ( xi ) with jε ( x ) : = { |x| , if |x| > ε , 1 2 ( x2 ε + ε ) , if |x| ≤ ε . ( 1 ) The function Jε can be considered as a scaled Huber loss function which is widely used in robust regression analysis [ 40 , 50 ] . Moreover , the function Jε is continuously differentiable and fulfills |x| ≤ jε ( x ) ≤ |x| + ε for each x ∈ R. Instead of minimizing the function Jε directly , the idea of IRLS is to minimize instead a suitable chosen quadratic function Qε ( · , x ) , which majorizes Jε such that Qε ( z , x ) ≥ Jε ( z ) for all z ∈ RN . This function is furthermore chosen such that Qε ( x , x ) = Jε ( x ) holds , which implies that min z∈Rn Qε ( z , x ) ≤ Jε ( x ) . The latter inequality implies that by minimizing Qε ( · , x ) , IRLS actually achieves an improvement in the value of Jε as well . More specifically , Qε ( · , x ) is defined by Qε ( z , x ) : = Jε ( x ) + 〈∇Jε ( x ) , z − x〉+ 1 2 〈 ( z − x ) , diag ( wε ( x ) ) ( z − x ) 〉 = Jε ( x ) + 1 2 〈z , diag ( wε ( x ) ) z〉 − 1 2 〈x , diag ( wε ( x ) ) x〉 , ( 2 ) where ∇Jε ( x ) = ( { xi |xi| , if |xi| > ε xi ε , if |xi| ≤ ε ) N i=1 is the gradient of Jε at x and the weight vector wε ( x ) ∈ RN is a vector of weights such that wε ( x ) i : = [ max ( |xi| , ε ) ] −1 for i ∈ [ N ] . The following lemma shows that Qε ( · , · ) has indeed the above-mentioned properties . We refer to the supplementary material for a proof . Lemma 2.1 . Let ε > 0 , let Jε : RN → R be defined as in ( 1 ) and Qε : RN ×RN → R as defined in ( 2 ) . Then , for any z , x ∈ RN , the following affirmations hold : i. diag ( wε ( x ) ) x = ∇Jε ( x ) , ii . Qε ( x , x ) = Jε ( x ) , iii . Qε ( z , x ) ≥ Jε ( z ) . As can be seen from the equality in ( 2 ) , minimizing Qε ( · , x ) corresponds to a minimization of a ( re- ) weighted least squares objective 〈· , diag ( wε ( x ) ) ·〉 , which lends its name to the method . Note that unlike a classical MM approach , however , IRLS comes with an update step of the smoothing parameter ε at each iteration . We provide an outline of the method in Algorithm 1 . The weighted least squares update ( 3 ) can be computed such that xk+1 = W−1k A ∗ ( AW−1k A ∗ ) −1 ( y ) with Wk = diag ( wk ) , with the solution of the ( m×m ) linear system ( AW−1k A∗ ) z = y as a main computational step . This linear system is positive definite and suitable for the use of iterative solvers . In [ 30 ] , an analysis of how accurately the linear system of a similar IRLS method needs to be solved to ensure overall convergence . We note that for small εk , the Sherman-Woodbury formula [ 70 ] can be used so that the calculation of xk+1 boils down to solving a smaller linear system that is well-conditioned , c.f . the supplementary material for details . This numerically advantageous property is not shared by the methods of [ 3 , 23 , 30 ] , as our smoothing update ( 4 ) is slightly different from the ones proposed in these papers . We refer to Section 2.2 for a discussion . The update step of the smoothing parameter ε ( 4 ) for the IRLS algorithm under consideration requires an a priori estimate of the sparsity of the ground truth of the signal , a piece of information that is Algorithm 1 Iteratively Reweighted Least Squares for ` 1-minimization Input : Measurement matrix A ∈ Rm×N , data vector y ∈ Rm , initial weight vector w0 ∈ RN ( default : w0 = ( 1 , 1 , . . . , 1 ) ) . Set ε0 =∞ . for k = 0 , 1 , 2 , . . . do xk+1 : = arg min z∈RN 〈z , diag ( wk ) z〉 subject to Az = y , ( 3 ) εk+1 : = min ( εk , σs ( x k+1 ) ` 1 N ) , ( 4 ) ( wk+1 ) i : = 1 max ( |xk+1i | , εk+1 ) for each i ∈ [ N ] , ( 5 ) end for return Sequence ( xk ) k≥1 . also needed by most of the methods for sparse reconstruction . In practice , an overestimation of s is not a problem for similar numerical results if the overestimation remains within small multiples of the sparsity of the signal . We note , however , that there are also versions of IRLS which do not require a-priori knowledge of s , e.g . [ 30,68 ] , as the update rule for the smoothing parameter is chosen differently . An interesting future research direction is to extend the analysis presented here to IRLS with such a smoothing parameter update . A consequence of Lemma 2.1 , step ( 4 ) , the fact that ε 7→ Jε ( z ) is monotonously non-decreasing , and that k 7→ εk is non-increasing is that k 7→ Jε ( z ) is non-increasing in k. This implies that the iterates xk , xk+1 of Algorithm 1 fulfill Jεk+1 ( xk+1 ) ≤ Jεk ( xk+1 ) ≤ Qεk ( xk+1 , xk ) ≤ Qεk ( xk , xk ) = Jεk ( xk ) . ( 6 ) This shows in particular that the sequence { Jεk ( xk ) } ∞ k=0 is non-increasing . For this reason , it can be shown that each accumulation point of the sequence of iterates ( xk ) k≥0 is a ( first-order ) stationary point of the smoothed ` 1-objective Jε ( · ) subject to the measurement constraint imposed by A and y , where ε = limk→∞ εk ( see [ 23 , Theorem 5.3 ] ) .

Summary: The paper studies the Basis Pursuit (BP) optimization problem (minimal l1 norm solution to a linear system). Iterative Reweighted Least Squares (IRLS) is a majorization minimization algorithm to solve this problem. The paper shows that it converges *globally* and linearly, under a nullspace property assumption (with parameter <1/2). Existing results only prove local linear convergence rates, first showing that IRLS identifies the support of the solution, after which the rate becomes linear. The key is a new update rule for the smoothing parameter used in the majorization of |.| by a scaled Huber function. Experiments highlight the two regimes : global convergence shown in this paper, followed by a linear convergence with a better rate once the iterate enter a basin of attraction of the solution.