Smooth Bilevel Programming for Sparse Regularization

1 Introduction . Regularized empirical risk minimization is a workhorse of supervised learning , and for a linear model , it reads min β∈Rn R ( β ) + 1 λ L ( Xβ , y ) ( Pλ ) whereX ∈ Rm×n is the design matrix ( n being the number of samples andm the number of features ) , L : Rm × Rm → [ 0 , ∞ ) is the loss function , and R : Rn → [ 0 , ∞ ) the regularizer . Here λ > 0 is the regularisation parameter which is typically tuned by cross-validation , and in the limit case λ = 0 , ( P0 ) is a constraint problem minβ R ( β ) under the constraint L ( Xβ , y ) = 0 . In this work , we focus our attention to sparsity enforcing penalties , which induce some form of structure on the solution of ( Pλ ) , the most celebrated examples ( reviewed in Section 2 ) being the Lasso , group-Lasso and trace norm regularizers . All these regularizers , and much more ( as detailed in Section ) , can be conveniently re-written as an infimum of quadratic functions . While Section 2 reviews more general formulations , this so-called “ quadratic variational form ” is especially simple in the case of block-separable functionals ( such as Lasso and group-Lasso ) , where one has R ( β ) = min η∈Rk+ 1 2 ∑ g∈G ||βg||22 ηg + 1 2 h ( η ) , ( 1 ) ∗Department of mathematical sciences , University of Bath , Bath BA2 7AY , UK cmshp20 @ bath.ac.uk †CNRS and DMA , Ecole Normale Supérieure , PSL University , 45 rue d ’ Ulm , F-75230 PARIS cedex 05 , FRANCE , gabriel.peyre @ ens.fr 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . where G is a partition of { 1 , . . . , n } , k = |G| is the number of groups and h : Rk+ → [ 0 , ∞ ) . An important example is the group-Lasso , where R ( β ) = ∑ g ||βg||2 is a group- ` 1 norm , in which case h ( η ) = ∑ i ηi . The special case of the Lasso , corresponding to the ` 1 norm is obtained when g = { i } for i = 1 , . . . , n and k = n. This quadratic variational form ( 1 ) is at the heart of the Iterative Reweighted Least Squares ( IRLS ) approach , reviewed in Section 1.1 . We refer to Section 2 for an in-depth exposition of these formulations . Sparsity regularized problems ( Pλ ) are notoriously difficult to solve , especially for small λ , because R is a non-smooth function . It is the non-smoothness of R which forces the solutions of ( Pλ ) to belong to low-dimensional spaces ( or more generally manifolds ) , the canonical example being spaces of sparse vectors when solving a Lasso problem . We refer to Bach et al . [ 2011 ] for an overview of sparsity-enforcing regularization methods . The core idea of our algorithm is that a simple reparameterization of ( 1 ) combined with a bi-level programming ( i.e . solving two nested optimization problems ) can turn ( Pλ ) into a smooth program which is much better conditioned , and can be tackled using standard but highly efficient optimization techniques such as quasi-Newton ( L-BFGS ) . Indeed , by doing the change of variable ( vg , ug ) , ( √ ηg , βg/ √ ηg ) in ( 1 ) , ( Pλ ) is equivalent to min v∈Rk f ( v ) where f ( v ) , min u∈Rn G ( u , v ) ( 2 ) G ( u , v ) , 1 2 h ( v v ) + 1 2 ||u||2 + 1 λ L ( X ( v G u ) , y ) . ( 3 ) Throughout , we define to be the standard Hadamard product and for v ∈ Rk and u ∈ Rn , we define v G u ∈ Rn to be such that ( v G u ) g = vgug . Provided that v 7→ h ( v v ) is differentiable and L ( · , y ) is a convex , proper , lower semicontinuous function , the inner minimisation problem has a unique solution and f is differentiable . Moreover , in the case of the quadratic loss L ( z , y ) , 12 ||z − y|| 2 2 , the gradient of f can be computed in closed form , by solving a linear system of dimension m or n. This paper is thus devoted to study the theoretical and algorithmic implications of this simple twist on the celebrated IRLS approach . Comparison with proximal gradient To provide some intuition about the proposed approach , the figure on the right contrasts the iterations of a gradient descent on f and of the iterative soft thresholding algorithm ( ISTA ) on the Lasso . We consider a random Gaussian matrix X ∈ R10×20 with λ = ||X > y||∞/10 . The ISTA trajectory is non-smooth when some feature crosses 0 . In particular , if a coefficient ( such as the red one on the figure ) is initialized with the wrong sign , it takes many iteration for ISTA to flip sign . In sharp contrast , the gradient flow of f does not exhibit such a singularity and exhibits a smooth geometric convergence . We refer to the appendix for an analysis of this phenomenon . Contributions Our main contribution is a new versatile algorithm for sparse regularization , which applies standard smooth optimization methods to minimize the function f in ( 2 ) . We first propose in Section 2 a generic class of regularizers R that enjoy a quadratic variational form . This section recaps existing results under a common umbrella and shows the generality of our approach . Section 3.1 then gathers the theoretical analysis of the method , and in particular the proof that while being non-convex , the function f has no local minimum and only “ ridable ” saddle points . As a result , one can guarantee convergence to a global minimum for many optimisation schemes , such as gradient descent with random perturbations [ Lee et al. , 2017 , Jin et al. , 2017 ] or trust region type methods [ Pascanu et al. , 2014 ] . Furthermore , for the case of the group Lasso , we show that f is an infinitely differentiable function with uniformly bounded Hessian . Consequently , standard solvers such as Newton ’ s method/BFGS can be applied and with a superlinear convergence guarantee . Section 4 performs a detailed numerical study of the method and benchmarks it against several popular competing algorithms for Lasso , group-Lasso and trace norm regularization . Our method is consistently amongst the best performers , and is in particular very efficient for small values of λ , and can even cope with the constrained case λ = 0 . 1.1 Related works . State of the art solvers for sparse optimisation Popular approaches to nonsmooth sparse optimisation include proximal based methods . The simplest instance is the Forward-Backward algorithm [ Lions and Mercier , 1979 , Daubechies et al. , 2004 ] which handles the case where L is a smooth function . There are many related inertial based acceleration schemes , such as FISTA [ Beck and Teboulle , 2009 ] and particularly effective techniques that leads to substantial speedups are the adaptive use of stepsizes and restarting strategies [ O ’ donoghue and Candes , 2015 ] . Other related approaches are proximal quasi-Newton and variable metric methods [ Combettes and Vũ , 2014 , Becker et al. , 2019 ] , which incorporate variable metrics into the quadratic term of the proximal operator . Another popular approach , particularly for the Lasso-like problems are coordinate descent schemes [ Friedman et al. , 2010 ] . These schemes are typically combined with support pruning schemes [ Ghaoui et al. , 2010 , Ndiaye et al. , 2017 , Massias et al. , 2018 ] . In the case where both the regularisation and loss terms are nonsmooth ( e.g . in the basis pursuit setting ) , typical solvers are the primal-dual [ Chambolle and Pock , 2011 ] , ADMM [ Boyd et al. , 2011 ] and Douglas-Rachford algorithms [ Douglas and Rachford , 1956 ] . Although these schemes are very popular due to their relatively low per iteration complexity , these methods have sublinear convergence rates in general , with linear convergence under strong convexity . The quadratic variational formulation and IRLS Quadratic variational formulations such as ( 1 ) have been exploited in many early computer vision works , such as Geiger and Yuille [ 1991 ] and Geman and Reynolds [ 1992 ] . One of the first theoretical results can be found in Geman and Reynolds [ 1992 ] , see also Black and Rangarajan [ 1996 ] which lists many examples . Our result in Theorem 1 can be seen as a generalisation of these early works . Norm regularizers of this form were introduced in Micchelli et al . [ 2013 ] , and further studied in the monograph Bach et al . [ 2011 ] under the name of subquadratic norms . The main algorithmic consequence of the quadratic variational formulation in the literature is the iterative reweighted least squares ( IRLS ) algorithm . When L ( z , y ) = 12 ||z − y|| 2 2 is the quadratic loss , a natural optimisation strategy is alternating minimising . However , due to the 1/ηg term , one needs to introduce some regularisation to ensure convergence . One popular approach is to add ε2 ∑ g η −1 g to the formulation ( 1 ) which leads to the IRLS algorithm [ Daubechies et al. , 2010 ] . The ε-term can be seen as a barrier to keep ηg positive which is reminiscent of interior point methods . The idea of IRLS is to do alternating minimisation over β and η , where the minimisation with respect to β is a least squares problem , and the minimisation with respect to η admits a closed form solution . A nuclear norm version of IRLS has been used in Argyriou et al . [ 2008 ] where an alternating minimisation algorithm was introduced . Finally , we remark that although nonconvex formulations for various low complexity regularizers have appeared in the literature , see for instance Rennie and Srebro [ 2005 ] , Hastie et al . [ 2015 ] , Mardani and Giannakis [ 2015 ] , Hoff [ 2017 ] for the case of the ` 1 and nuclear norms , they are typically associated with alternating minimisation algorithms . Variable projection/reduced gradient approaches IRLS methods are quite slow because the resulting minimization problem is poorly conditioned . Adding the smoothing term ε2 ∑ g η −1 g only partly alleviates this , and also breaks the sparsity enforcing property of the regularizer R. We avoid both issues in ( 2 ) by solving a “ reduced ” problem which is much better conditioned and smooth . This idea of solving a bi-variate problem by re-casting it as a bilevel program is classical , we refer in particular to [ Rockafellar and Wets , 2009 , Chap . 10 ] for some general theoretical results on reduced gradients , and also Danskin ’ s theorem ( although this is restricted to the convex setting ) in Bertsekas [ 1997 ] . Our formulation falls directly into the framework of variable projection [ Ruhe and Wedin , 1980 , Golub and Pereyra , 2003 ] , introduced initially for solving nonlinear least squares problems . Properties and advantages of variable projection have been studied in Ruhe and Wedin [ 1980 ] , we refer also to Hong et al . [ 2017 ] , Zach and Bourmaud [ 2018 ] for more recent studies . Nonsmooth variable projection is studied in van Leeuwen and Aravkin [ 2016 ] , although the present work is in the classical setting of variable projection due to our smooth reparametrization . Reduced gradients have also been associated with the quadratic variational formulation in several works [ Bach et al. , 2011 , Pong et al. , 2010 , Rakotomamonjy et al. , 2008 ] . The idea is to apply descent methods over g ( η ) = minβ R0 ( η , β ) + 1 2 ||Xβ − y|| 2 2 . Although the function over η and β is discontinuous , the function g over η is smooth and one can apply first order methods , such as proximal gradient descent to minimise g under positivity constraints . While quasi-Newton methods can be applied in this setting with bound constraints , we show in Section 4.1 that this approach is typically less effective than our nonconvex bilevel approach . In the setting of the trace norm , the optimisation problem is constrained on the set of positive semidefinite matrices , so one is restricted to using first order methods [ Pong et al. , 2010 ] .

Summary: The authors study regularized regression with a sparsity inducing constraint. (In the appendix also the extension to the low-rankness inducing case is considered.) The authors propose a new non-convex formulation, which is optimized via BFGS. The non-convex formulation is motivated by the well-known IRLS (Iteratively Reweighted Least Squares) algorithm. Extensive numerical simulations are conducted on a wide variety of problems and the proposed algorithm is compared with a number of state-of-the-art approaches. While the proposed algorithm does not always perform best, it is in all cases at least competitive to the best performing approach. Moreover, the authors prove that the induced non-convex function, which is optimized in their approach, is ridable -- meaning that all local minima are global minima and the Hessians of all saddle points have at least one negative eigenvalue.