Fully Polynomial-Time Randomized Approximation Schemes for Global Optimization of High-Dimensional Folded Concave Penalized Generalized Linear Models

1 INTRODUCTION . This paper concerns global optimization of a folded concave penalized formulation of highdimensional learning generalized linear models , which belongs to statistical/machine learning problems such that the number of dimensions ( or number of fitting parameters ) p is ( much ) larger than the number of samples n. This type of problem has recently become very common in a variety of engineering and scientific applications ( Fan et al. , 2014a ; Fan & Li , 2006 ) including computational biology , speech recognition and image processing ( Huang et al. , 2016 ; Algamal & Lee , 2015 ; Xing et al. , 2001 ; Palmer & Ostendorf , 2001 ; Bagaria et al. , 2012 ; Paul et al. , 2013 ) . Globally minimal solutions to such a nonconvex learning formulation have been shown effective to guarantee desirable statistical performance in order to address high dimensionality ( Zhang & Zhang , 2012 ) . Nonetheless , generating a global solution admits no pseudo polynomial-time algorithm , unless “ P = NP ” ; Indeed , global optimality is shown strongly NP-hard to achieve by ( Chen et al. , 2017 ; Huo & Chen , 2010 ) while Chen et al . ( 2014 ) show similar results for several related problems in regularized minimization . In contrast to the existing pessimistic result , we derive herein a fully polynomial-time randomized approximation scheme ( FPRAS , as defined in 4 ) that theoretically ensures global minimality to 2 with high probability . Specifically , consider a high-dimensional generalized linear model ( GLM ) as follows . Let X = ( x1 , ... , xn ) ᵀ be the n×p design matrix with xi = ( xi1 , ... , xip ) ᵀ , i = 1 , ... , n , and Y = ( y1 , ... , yn ) ᵀ be the n-dimensional response vector . We will assume the design matrix X is fixed , while the mean of the response is given by E [ yi ] = ψ′ ( x > i β true ) for some known link function ψ : Θ→ < , where Θ ⊆ < and βtrue = ( βtrue1 , ... , βtruep ) is the unknown vector of true parameters of the model . Such a setup can be seen as a generalization of linear regression models with the link function allowing for nonlinear transformations that enable a more flexible approach to model estimation . The high-dimensional regression problem is to estimate βtrue given knowledge of X , Y , and ψ in the undesirable scenario where p n > 0 . To that end , traditional statistical learning schemes often resort to the following formulation : L ( β ) = n∑ i=1 ` ( yi , xi , β ) = 1 n n∑ i=1 [ ψ ( xᵀi β ) − yix ᵀ i β ] , ( 1 ) which , according to traditional statistical theories , would result in over fitting in general under the high-dimensional setting . To resolve over fitting , modern statistical theories favor a modified formulation as below : min β Q ( β ) : = L ( β ) + p∑ j=1 Pλ ( |βj | ) , ( 2 ) where Pλ ( | · | ) is sparsity-inducing regularization term that penalizes any nonzero dimensions in the minimizer , and λ > 0 is a tuning parameter . Under the assumption that the true regression parameter βtrue is sparse , a global optimizer to equation 2 has been shown effective to address over fitting for many choices of specific regularization functions Pλ . Indeed , one of the most successful choices of Pλ is the much studied Lasso-based regularized ( Tibshirani , 1996 ) , aka , the ` 1 ( -norm ) penalty , which has been demonstrated to entail desirable statistical properties ( Bickel et al. , 2009 ; Negahban et al. , 2012 ) . Another admirable property of the Lasso is that , especially when applied to least squares linear regression , it yields an extremely tractable problem via a variety of algorithms ( Friedman et al. , 2008 ; 2010 ) . However , per ( Zhao & Yu , 2006 ; Fan & Li , 2001 ) , Lasso is not selection consistent without a strong irrepresentable condition and may sometimes introduce nontrivial estimation bias . As a popular alternative to Lasso , the folded concave penalty ( FCP ) was first introduced by ( Fan & Li , 2001 ) . There are mainstream examples of FCP functions , including the SCAD by ( Fan & Li , 2001 ) and MCP by ( Zhang , 2010 ) . This paper will focus on MCP , defined as Pλ ( |t| ) =∫ |t| 0 ( aλ−s ) + a ds for some fixed parameter a > 0 . In contrast to the Lasso , the FCP regulaizations achieve variable selection consistency non-contingent on the irrepresentable condition and is demonstrated to be unbiased ( Fan & Li , 2001 ) . Furthermore , ( Zhang & Zhang , 2012 ) demonstrated that the global solution to the FCP-regularized formulation leads to desirable recovery of the oracle solution . Nonetheless , FCP problems are significantly harder to solve than Lasso , the new penalty term moves the problem outside the realm of convex optimization , ( Chen et al. , 2017 ) even showed that any estimation problem with convex loss and folded concave regularization to be strongly NP-hard , ruling out the possibility of a pseudo-polynomial-time global optimization algorithm . ( Liu et al. , 2016 ) were seemingly the first to propose a global approach to the problem called MIPGO which reformulates the problem into a mixed integer program . Yet , the worst-case complexity of MIPGO is in exponential time . Perhaps for this reason , current literature tends to focus on local algorithms for the FCP-regularized learning problems . The local quadratic approximation algorithm by ( Fan & Li , 2001 ) is an example of a majorization minimization algorithm , an approach which is also related to the local linear approximation ( LLA ) algorithm proposed by ( Zou & Li , 2008 ) . LLA was further explored by ( Fan et al. , 2014b ) showing the oracle property can be obtained with high probability despite the local approach . ( Mazumder et al. , 2011 ; Fan & Lv , 2011 ) demonstrate coordinate optimization approaches for FCP while ( Wang et al. , 2014 ) used an approximate regularization path-following algorithm to obtain the optimal convergence rate to statistically desirable local solution . ( Wang et al. , 2013 ) analyzed the CCCP algorithm and showed under certain conditions that it asymptotically finds the oracle estimator . ( Liu et al. , 2017 ) took an algorithm agnostic approach by analyzing local solutions satisfying second order KKT conditions and showed desirable statistical properties like recovering the oracle solution and sparisty . These results discussed above primarily relate to FCP-regularized linear regression , a special case of GLM where ψ is specifically the identity function . For analyses which encompass GLM ’ s with FCP regularizers , ( Fan & Lv , 2011 ) showed that GLM ’ s , even in ultra high dimensional variable selection problems , have oracle properties when using FCP regularization and demonstrated a coordinate optimization algorithm for finding local solutions . In the area of Mestimators , which is a further generalization of our estimation method beyond even GLMs , ( Loh & Wainwright , 2013 ; Loh et al. , 2017a ) showed that under certain conditions all local solutions must be within statistical precision of the true parameter and its support while ( Loh et al. , 2017b ) demonstrate a two-step algorithm involving composite gradient descent to find a local solution . Bian & Chen ( 2014 ) demonstrate a optimality conditions for a class of nonconvex optimization problems using nonlipscitz regularization . From the numerous results pertaining to local solution schemes above , our research question is why local solutions are repetitively successful . In other words , are there certain geometric properties of the learning formulation equation 2 with FCP that allow all local schemes to perform well independent of the specific designs of the algorithmic procedures ? Our answer to this question is affirmative ; we show herein that all local solutions within an efficiently achievable sub-level set are actually globally optimal . Those local solutions are characterized by the significant subspace second-order necessary conditions ( S3ONC ) provable admit FPRAS ’ s . The S3ONC are weaker conditions than the standard second-order KKT conditions . As per this result , all S3ONC-guaranteeing algorithms ( which include a large spectrum of local algorithms ) belong to the class of FPRAS ’ s for global optimization of the strongly NP-hard FCP-based learning problem . We subsequently develop theories for two specific algorithms of this type : one gradient-based method and the other is the same as the LLA . It is worth noting that ( Zhang , 2010 ) provides conditions to establish the uniqueness of local solutions to FCP-based learning . When local solutions are unique , then any local optimization algorithms would ensure global optimality . However , a few critical assumptions are necessary to achieve the uniqueness result and , furthermore , many report numerical experiments , e.g. , those in ( Fan et al. , 2014b ; Liu et al. , 2017 ; 2016 ; Fan & Li , 2001 ) indicate the non-uniqueness of local solutions , instead . In contrast , our results in this paper imposes only standard assumptions commonly shared by a flexible set of high-dimensional GLMs and are applicable even if the local solutions are non-unique . To our knowledge , this is the first geometric proof that global solutions coincide with pseudo-polynomial-time computable local solutions in an FCP-based regression formulation with high probability . The resulting algorithms are the first few FPRAS ’ s to this problem . Two works with notable relations to our own are Liu et al . ( 2017 ) and Liu & Ye ( 2019 ) . The first applies a similar analytical framework to linear regression problems however our generalization to GLMs adds significant flexibility and it was unknown for their result that the oracle solution implies global optimality since it was only as of Zhang & Zhang ( 2012 ) that global optimality was known to potentially imply the oracle solution . Further , it is nontrivial to extend their existing result to global optimal results . ( Liu & Ye , 2019 ) on the other hand is a more general setup than our own though the tradeoff is that our rate is better and S3ONC solutions do not ensure global optimality to the in-sample training error for their setup . The rest of this paper is organized as follows . Section 2 goes through specific problem assumptions and explains the S3ONC . Section 3 contains our main result for global optimality and uses it to make additional claims for LLA . Section 4 contains numerical results to verify our theoretical findings . Section 5 provides concluding remarks . In this paper we will use ‖·‖0 to denote the number of nonzero entries , | · | to denote the ` 1-norm if the argument is a vector , or cardinality if the argument is a set , ‖·‖ to denote the ` 2-norm , ‖·‖max to denote the maximum norm and ‖·‖min to denote the absolute value of the entry with the smallest magnitude . ( · ) + is used equivalently to max ( 0 , · ) . For any vector v , vQ is intended as ( vj : j ∈ Q ) . For any set Q , we denote the complement as Qc . In particular , let S be the true support set , that is , S : = { j : βtruej 6= 0 } and its complement is Sc . We occasionally use the term the “ oracle solution ” to refer to the solution βoracle defined as βoracle ∈ arg min β : βj=0 , ∀j /∈S L ( β ) . The oracle solution is a hypothetical solution which assumes prior knowledge of the true support set S and thus can be considered a theoretical benchmark .

Summary: The authors study the high-dimensional sparse estimation problems, which is one of the fundamental topics in both machine learning and optimization communities. In the literature, the folded concave penalty (FCP) methods have been shown to enjoy the strong oracle property for high-dimensional sparse estimation.  While LASSO solutions can be easily computed, such solutions do not admit unbiasedness and oracle property or require strong conditions. Therefore,  the problems with FCP have received much attentions and been studied in terms of hardness and approximability of the problems recently .