Smooth Bilevel Programming for Sparse Regularization

1 Introduction . Regularized empirical risk minimization is a workhorse of supervised learning , and for a linear model , it reads min β∈Rn R ( β ) + 1 λ L ( Xβ , y ) ( Pλ ) whereX ∈ Rm×n is the design matrix ( n being the number of samples andm the number of features ) , L : Rm × Rm → [ 0 , ∞ ) is the loss function , and R : Rn → [ 0 , ∞ ) the regularizer . Here λ > 0 is the regularisation parameter which is typically tuned by cross-validation , and in the limit case λ = 0 , ( P0 ) is a constraint problem minβ R ( β ) under the constraint L ( Xβ , y ) = 0 . In this work , we focus our attention to sparsity enforcing penalties , which induce some form of structure on the solution of ( Pλ ) , the most celebrated examples ( reviewed in Section 2 ) being the Lasso , group-Lasso and trace norm regularizers . All these regularizers , and much more ( as detailed in Section ) , can be conveniently re-written as an infimum of quadratic functions . While Section 2 reviews more general formulations , this so-called “ quadratic variational form ” is especially simple in the case of block-separable functionals ( such as Lasso and group-Lasso ) , where one has R ( β ) = min η∈Rk+ 1 2 ∑ g∈G ||βg||22 ηg + 1 2 h ( η ) , ( 1 ) ∗Department of mathematical sciences , University of Bath , Bath BA2 7AY , UK cmshp20 @ bath.ac.uk †CNRS and DMA , Ecole Normale Supérieure , PSL University , 45 rue d ’ Ulm , F-75230 PARIS cedex 05 , FRANCE , gabriel.peyre @ ens.fr 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . where G is a partition of { 1 , . . . , n } , k = |G| is the number of groups and h : Rk+ → [ 0 , ∞ ) . An important example is the group-Lasso , where R ( β ) = ∑ g ||βg||2 is a group- ` 1 norm , in which case h ( η ) = ∑ i ηi . The special case of the Lasso , corresponding to the ` 1 norm is obtained when g = { i } for i = 1 , . . . , n and k = n. This quadratic variational form ( 1 ) is at the heart of the Iterative Reweighted Least Squares ( IRLS ) approach , reviewed in Section 1.1 . We refer to Section 2 for an in-depth exposition of these formulations . Sparsity regularized problems ( Pλ ) are notoriously difficult to solve , especially for small λ , because R is a non-smooth function . It is the non-smoothness of R which forces the solutions of ( Pλ ) to belong to low-dimensional spaces ( or more generally manifolds ) , the canonical example being spaces of sparse vectors when solving a Lasso problem . We refer to Bach et al . [ 2011 ] for an overview of sparsity-enforcing regularization methods . The core idea of our algorithm is that a simple reparameterization of ( 1 ) combined with a bi-level programming ( i.e . solving two nested optimization problems ) can turn ( Pλ ) into a smooth program which is much better conditioned , and can be tackled using standard but highly efficient optimization techniques such as quasi-Newton ( L-BFGS ) . Indeed , by doing the change of variable ( vg , ug ) , ( √ ηg , βg/ √ ηg ) in ( 1 ) , ( Pλ ) is equivalent to min v∈Rk f ( v ) where f ( v ) , min u∈Rn G ( u , v ) ( 2 ) G ( u , v ) , 1 2 h ( v v ) + 1 2 ||u||2 + 1 λ L ( X ( v G u ) , y ) . ( 3 ) Throughout , we define to be the standard Hadamard product and for v ∈ Rk and u ∈ Rn , we define v G u ∈ Rn to be such that ( v G u ) g = vgug . Provided that v 7→ h ( v v ) is differentiable and L ( · , y ) is a convex , proper , lower semicontinuous function , the inner minimisation problem has a unique solution and f is differentiable . Moreover , in the case of the quadratic loss L ( z , y ) , 12 ||z − y|| 2 2 , the gradient of f can be computed in closed form , by solving a linear system of dimension m or n. This paper is thus devoted to study the theoretical and algorithmic implications of this simple twist on the celebrated IRLS approach . Comparison with proximal gradient To provide some intuition about the proposed approach , the figure on the right contrasts the iterations of a gradient descent on f and of the iterative soft thresholding algorithm ( ISTA ) on the Lasso . We consider a random Gaussian matrix X ∈ R10×20 with λ = ||X > y||∞/10 . The ISTA trajectory is non-smooth when some feature crosses 0 . In particular , if a coefficient ( such as the red one on the figure ) is initialized with the wrong sign , it takes many iteration for ISTA to flip sign . In sharp contrast , the gradient flow of f does not exhibit such a singularity and exhibits a smooth geometric convergence . We refer to the appendix for an analysis of this phenomenon . Contributions Our main contribution is a new versatile algorithm for sparse regularization , which applies standard smooth optimization methods to minimize the function f in ( 2 ) . We first propose in Section 2 a generic class of regularizers R that enjoy a quadratic variational form . This section recaps existing results under a common umbrella and shows the generality of our approach . Section 3.1 then gathers the theoretical analysis of the method , and in particular the proof that while being non-convex , the function f has no local minimum and only “ ridable ” saddle points . As a result , one can guarantee convergence to a global minimum for many optimisation schemes , such as gradient descent with random perturbations [ Lee et al. , 2017 , Jin et al. , 2017 ] or trust region type methods [ Pascanu et al. , 2014 ] . Furthermore , for the case of the group Lasso , we show that f is an infinitely differentiable function with uniformly bounded Hessian . Consequently , standard solvers such as Newton ’ s method/BFGS can be applied and with a superlinear convergence guarantee . Section 4 performs a detailed numerical study of the method and benchmarks it against several popular competing algorithms for Lasso , group-Lasso and trace norm regularization . Our method is consistently amongst the best performers , and is in particular very efficient for small values of λ , and can even cope with the constrained case λ = 0 . 1.1 Related works . State of the art solvers for sparse optimisation Popular approaches to nonsmooth sparse optimisation include proximal based methods . The simplest instance is the Forward-Backward algorithm [ Lions and Mercier , 1979 , Daubechies et al. , 2004 ] which handles the case where L is a smooth function . There are many related inertial based acceleration schemes , such as FISTA [ Beck and Teboulle , 2009 ] and particularly effective techniques that leads to substantial speedups are the adaptive use of stepsizes and restarting strategies [ O ’ donoghue and Candes , 2015 ] . Other related approaches are proximal quasi-Newton and variable metric methods [ Combettes and Vũ , 2014 , Becker et al. , 2019 ] , which incorporate variable metrics into the quadratic term of the proximal operator . Another popular approach , particularly for the Lasso-like problems are coordinate descent schemes [ Friedman et al. , 2010 ] . These schemes are typically combined with support pruning schemes [ Ghaoui et al. , 2010 , Ndiaye et al. , 2017 , Massias et al. , 2018 ] . In the case where both the regularisation and loss terms are nonsmooth ( e.g . in the basis pursuit setting ) , typical solvers are the primal-dual [ Chambolle and Pock , 2011 ] , ADMM [ Boyd et al. , 2011 ] and Douglas-Rachford algorithms [ Douglas and Rachford , 1956 ] . Although these schemes are very popular due to their relatively low per iteration complexity , these methods have sublinear convergence rates in general , with linear convergence under strong convexity . The quadratic variational formulation and IRLS Quadratic variational formulations such as ( 1 ) have been exploited in many early computer vision works , such as Geiger and Yuille [ 1991 ] and Geman and Reynolds [ 1992 ] . One of the first theoretical results can be found in Geman and Reynolds [ 1992 ] , see also Black and Rangarajan [ 1996 ] which lists many examples . Our result in Theorem 1 can be seen as a generalisation of these early works . Norm regularizers of this form were introduced in Micchelli et al . [ 2013 ] , and further studied in the monograph Bach et al . [ 2011 ] under the name of subquadratic norms . The main algorithmic consequence of the quadratic variational formulation in the literature is the iterative reweighted least squares ( IRLS ) algorithm . When L ( z , y ) = 12 ||z − y|| 2 2 is the quadratic loss , a natural optimisation strategy is alternating minimising . However , due to the 1/ηg term , one needs to introduce some regularisation to ensure convergence . One popular approach is to add ε2 ∑ g η −1 g to the formulation ( 1 ) which leads to the IRLS algorithm [ Daubechies et al. , 2010 ] . The ε-term can be seen as a barrier to keep ηg positive which is reminiscent of interior point methods . The idea of IRLS is to do alternating minimisation over β and η , where the minimisation with respect to β is a least squares problem , and the minimisation with respect to η admits a closed form solution . A nuclear norm version of IRLS has been used in Argyriou et al . [ 2008 ] where an alternating minimisation algorithm was introduced . Finally , we remark that although nonconvex formulations for various low complexity regularizers have appeared in the literature , see for instance Rennie and Srebro [ 2005 ] , Hastie et al . [ 2015 ] , Mardani and Giannakis [ 2015 ] , Hoff [ 2017 ] for the case of the ` 1 and nuclear norms , they are typically associated with alternating minimisation algorithms . Variable projection/reduced gradient approaches IRLS methods are quite slow because the resulting minimization problem is poorly conditioned . Adding the smoothing term ε2 ∑ g η −1 g only partly alleviates this , and also breaks the sparsity enforcing property of the regularizer R. We avoid both issues in ( 2 ) by solving a “ reduced ” problem which is much better conditioned and smooth . This idea of solving a bi-variate problem by re-casting it as a bilevel program is classical , we refer in particular to [ Rockafellar and Wets , 2009 , Chap . 10 ] for some general theoretical results on reduced gradients , and also Danskin ’ s theorem ( although this is restricted to the convex setting ) in Bertsekas [ 1997 ] . Our formulation falls directly into the framework of variable projection [ Ruhe and Wedin , 1980 , Golub and Pereyra , 2003 ] , introduced initially for solving nonlinear least squares problems . Properties and advantages of variable projection have been studied in Ruhe and Wedin [ 1980 ] , we refer also to Hong et al . [ 2017 ] , Zach and Bourmaud [ 2018 ] for more recent studies . Nonsmooth variable projection is studied in van Leeuwen and Aravkin [ 2016 ] , although the present work is in the classical setting of variable projection due to our smooth reparametrization . Reduced gradients have also been associated with the quadratic variational formulation in several works [ Bach et al. , 2011 , Pong et al. , 2010 , Rakotomamonjy et al. , 2008 ] . The idea is to apply descent methods over g ( η ) = minβ R0 ( η , β ) + 1 2 ||Xβ − y|| 2 2 . Although the function over η and β is discontinuous , the function g over η is smooth and one can apply first order methods , such as proximal gradient descent to minimise g under positivity constraints . While quasi-Newton methods can be applied in this setting with bound constraints , we show in Section 4.1 that this approach is typically less effective than our nonconvex bilevel approach . In the setting of the trace norm , the optimisation problem is constrained on the set of positive semidefinite matrices , so one is restricted to using first order methods [ Pong et al. , 2010 ] .

Summary: The authors propose a bilevel optimization approach for solving regularized empirical risk minimization. The considered minimization problem ($\cal{P}_{\lambda}$) refers to a linear model with parameters $\beta$ that are regularized in terms of a sparsity enforcing term $R(\beta)$. More specifically, the authors treat regularizers $R$ that can be recast in a so-called quadratic variational form. Therewith, a reparametrization of the original problem can be regarded as a bilevel program with smooth inner problem. Subsequent to the introduction, the authors provide a characterization of regularizers that have a quadratic variational form, followed by a theoretical analysis and numerical experiments.