Randomized Primal-Dual Coordinate Method for Large-scale Linearly Constrained Nonsmooth Nonconvex Optimization

The large-scale linearly constrained nonsmooth nonconvex optimization finds wide applications in machine learning , including non-PSD Kernel SVM , linearly constrained Lasso with nonsmooth nonconvex penalty , etc . To tackle this class of optimization problems , we propose an efficient algorithm called Nonconvex Randomized Primal-Dual Coordinate ( N-RPDC ) method . At each iteration , this method only randomly selects a block of primal variables to update rather than updating all the variables , which is suitable for large-scale problems . We provide two types of convergence results for N-RPDC . We first show that any cluster point of the sequence of iterates generated by N-RPDC is almost surely ( i.e. , with probability 1 ) a stationary point . In addition , we also provide an almost sure asymptotic convergence rate of O ( 1/ √ k ) . Next , we establish the expected O ( ε−2 ) iteration complexity of N-RPDC in order to drive a natural stationarity measure below ε in expectation . The fundamental aspect to establishing the aforementioned convergence results is a surrogate stationarity measure we discovered for analyzing N-RPDC . Finally , we conduct a set of experiments to show the efficacy of NRPDC . 1 INTRODUCTION . Many large scale problems arising in machine learning amounts to solving the following linearly constrained nonsmooth nonconvex optimization problem : min x∈X F ( x ) = f ( x ) + g ( x ) s.t Ax− b = 0 , ( P ) where A ∈ Rn×d and b ∈ Rn . Throughout this paper , we impose the following assumptions on problem ( P ) : the function f : Rd → R is possibly nonconvex and continuously differentiable with its gradient ∇f being Lf -Lipschitz continuous , X is a convex and compact set of Rd , i.e. , there exits a positive number M such that M = maxx , x′∈X ‖x− x′‖ , and g : Rd → R is nonsmooth and nonconvex . More precisely , g is assumed to be lower semicontinuous ( l.s.c ) ρg-weakly convex and has bounded subgradients over X , i.e. , there exists Lg > 0 such that ‖s‖ ≤ Lg , ∀s ∈ ∂g ( x ) , x ∈ X , where ∂g is the subdifferential of g ( see ( 4 ) for definition ) . Let F ∗ be the optimal value of ( P ) . Recall that a function g is said to be ρg-weakly convex if g ( · ) + ρg2 ‖ · ‖ 2 is convex for some constant ρg ≥ 0 ( Vial , 1983 ) . It is worth mentioning that a wide class of nonsmooth nonconvex functions belong to the weakly convex class ; see , e.g. , ( Vial , 1983 ; Davis & Drusvyatskiy , 2019 ) for more discussions on weak convexity . We further assume g ( x ) = ∑N i=1 gi ( xi ) is block separable with respect to the space decomposition X = X1 × X2 × · · · × XN , xi ∈ Xi ⊂ Rdi , and N∑ i=1 di = d. ( 1 ) Let A = ( A1 , A2 , . . . , AN ) ∈ Rn×d be the corresponding partition of A , where Ai ∈ Rn×di . When N = d , each Xi ⊂ R corresponds to a box constraint on the coordinate xi for i = 1 , . . . , d. Motivations . Our motivation for studying large scale problem ( P ) stems from the fact that this problem class finds wide applications in machine learning . To be more specific , we will present several practical examples below that give rise to ( P ) . Application 1 : Non-PSD kernel support vector machine . Support vector machine ( SVM ) is a widely utilized technique for supervised learning ( Boser et al. , 1992 ; Cortes & Vapnik , 1995 ) . In order to improve the interpretability and enhance the robustness , some non-PSD kernels are used in the kernalized SVM method , such as the sigmoid kernel ( Lin & Lin , 2003 ) , the jittering kernel ( DeCoste & Schölkopf , 2002 ) , the tangent distance kernel ( Haasdonk & Keysers , 2002 , August ) , to name a few . Formally , the kernalized SVM problem can be written as min x∈ [ 0 , c ] d 1 2x > Qx− 1 > d x s.t . y > x = 0 , ( 2 ) where Q is a d× d non-PSD matrix , c ∈ R is the upper bound of all variables , y ∈ { −1 , 1 } d is the vector of labels , and 1d is an d-dimensional vector of all 1s . Application 2 : Linearly constrained Lasso with nonsmooth nonconvex penalty . The Lasso is one of the most popular methods for variable selection . The standard Lasso uses ` 1-norm to select important variables . Some works propose to utilize nonsmooth nonconvex regularizers to further improve the performance of the Lasso ; see , e.g. , ( Breheny & Huang , 2011 ; 2015 ; Rakotomamonjy et al. , 2019 ) . If additional prior information is available , the works ( Deng et al. , 2020 , May ; Gaines et al. , 2018 ; James et al. , 2013 ; Won et al. , 2019 ) propose the constrained Lasso model . The linearly constrained LASSO with nonsmooth nonconvex penalty can be formulated as follows : min x∈X 1 2‖Ax− b‖ 2 + d∑ i=1 φ ( xi ) s.t . Bx− c = 0 ( 3 ) where x = ( x1 , ... , xd ) > , X ⊂ Rd , A ∈ Rn×d is the design matrix , b ∈ Rn is the response vector , B ∈ Rm×d and c ∈ Rm are given constraints , and φ is a nonsmooth nonconvex regularizer such as MCP ( Zhang et al. , 2010 ) or SCAD ( Fan & Li , 2001 ) penalty . There are plenty of other applications that give rise to ( P ) , e.g. , the robust M-estimators , distributed learning , etc . Due to the limitation of space , we will not expand them in details here . Related works . Linearly constrained nonconvex optimization . Perhaps the most widely utilized class of algorithms for solving constrained optimization problems are the Primal-dual methods . Previous works on single-loop primal-dual methods mainly consider linearly contained convex optimization problems . The work ( Boţ & Nguyen , 2020 ) considers a nonconvex instance with a special linear constraint Ax− z = 0 , where both x and z are decision variables and the matrix A is assumed to be surjective . Under these assumptions , convergence results were established for a proximal alternating direction method of multipliers ( ADMM ) . The recent works ( Zhang & Luo , 2020a ; b ) provide convergence results of ADMM using gradient-based updates for linearly constrained smooth nonconvex optimization problems with a general linear constraint Ax = b , i.e. , problem ( P ) with g ≡ 0 . The main insights of their results are a construction of an proximally regularized auxiliary problem that dates back to ( Bertsekas , 1979 ) and a properly constructed Lyapunov function . Then , they established descent property on the Lyapunov function based on the dual error bound condition ( see also ( Hong & Luo , 2017 ) ) , which leads to iteration complexity results . The method introduced in ( Zhu et al. , 2020 ) can be used to solve problem ( P ) with specific linear constraints . This method is based on the auxiliary problem principle of augmented Lagrangian ( APP-AL ) method ( Cohen & Zhu , 1984 ; Zhao & Zhu , 2019 ; 2021 ) , which can be seen as a forward-backward splitting method and applies a Jacobian updating strategy.1 However , both ADMM-type and APP-AL-type methods compute the full gradient of primal variables and update all primal variables at each iteration . Therefore , the computation complexity of one iteration of these two types of methods are expensive for large-scale problems ( See Remark 2.1 ) . Randomized coordinate methods for unconstrained optimization . In the past decade , big data applications are ubiquitous in machine learning . The formulated optimization problems often involve very large datasets , and hence computing the function value or the gradient can be very expensive . These 1ADMM applies a Gauss-Seidel like minimization strategy , which is related to the Douglas-Rachford splitting method . observations motivate the study of the randomized coordinate methods . The researches on this type of methods can be traced back to ( Nesterov , 2012 ; 2014 ) . In ( Nesterov , 2012 ) , Nesterov studied the iteration complexity of the randomized coordinate gradient ( RCG ) descent method for smooth convex optimization . Later , in ( Nesterov , 2014 ) , the same author analyzed the randomized coordinate subgradient method for a set of piece-wise linear nonsmooth convex optimization problems . The works ( Richtárik & Takáč , 2014 ) and ( Lu & Xiao , 2015 ) extends Nesterov ’ s results to convex additive composite optimization . Iteration complexity and linear convergence of RCG were studied in ( Patrascu & Necoara , 2015 ) and ( Karimi et al. , 2016 ) for nonsmooth nonconvex optimization and smooth nonconvex optimization , respectively . Moreover , by employing an asynchronous point to evaluate the gradient in each iteration , The works ( Liu & Wright , 2015 ) and ( Liu et al. , 2014 ) establish the iteration complexity results of asynchronous RCG for convex smooth optimization and additive composite optimization , respectively . A Few results on randomized coordinate methods for linearly constrained optimization . Though the randomized coordinate-type methods for unconstrained optimization are extensively studied , there are only a few results for this type of methods concerning coupled linearly constrained optimization problems . The works ( Gao et al. , 2019 ; Wang et al. , 2014 ) study the randomized primal-dual coordinate ( RPDC ) method and provides iteration complexity results for linearly constrained convex optimization . Better complexity results of RPDC were obtained for linearly constrained strongly convex optimization ( Xu & Zhang , 2018 ) . The recent works ( Zhu & Zhao , 2020 ; Latafat et al. , 2019 ; Fercoq & Bianchi , 2019 ; Alacaoglu et al. , 2020 ) provide almost sure ( i.e. , with probability 1 ) asymptotic convergence results of RPDC-type methods for linearly constrained convex optimization . It is worth emphasizing that the works ( Latafat et al. , 2019 ; Fercoq & Bianchi , 2019 ; Alacaoglu et al. , 2020 ) establish sequential convergence results ( i.e. , the convergence of the whole sequence of iterates ) . However , the above mentioned works only considers either ADMM-type and APP-AL-type methods for linearly constrained smooth nonconvex optimization problem or the randomized coordinate methods for linearly constrained convex optimization . To the best of our knowledge , no previous result concerns convergence and iterate complexity of the randomized coordinate methods for linearly constrained nonsmooth nonconvex optimization , which will be the main focus of this paper . Main contributions . In this paper , we aim to solve the large-scale linearly constrained nonsmooth nonconvex optimization problem ( P ) . To tackle it , we propose the Nonconvex Randomized PrimalDual Coordinate ( N-RPDC ) method based on an auxiliary problem ( see Section 2 ) . Relying on the mild local uniform metric subregularity property ( see Section 3.3 ) , we provide two types of convergence results for N-RPDC ( see Section 4 ) . We first show that any cluster point of the sequence of iterates generated by N-RPDC is almost surely ( i.e. , with probability 1 ) a stationary point of ( P ) . In addition , we also provide an almost sure asymptotic convergence rate of O ( 1/ √ k ) , where k represents iteration number . Next , we establish the expected iteration complexity of N-RPDC . Namely , N-RPDC needs at most O ( ε−2 ) number of iterations in order to drive a surrogate stationarity measure below ε in expectation . The fundamental aspect to our convergence analysis is a surrogate stationarity measure we discovered for analyzing N-RPDC ( see Section 3.1 ) , which is one of the main contributions of this work . The standard stationarity measure ( i.e. , checking KKT conditions ) is not directly applicable here due to the stochastic nature of N-RPDC . Instead , we define the notion of reference point and use its distance to the iterate generated by N-RPDC as a surrogate stationarity measure . Then , such a surrogate stationarity measure is clarified by showing that it is an upper bound of the standard stationarity measure at the reference point up to a numerical constant ( see Proposition 3.1 and Remark 3.1 ) . There are also some other interesting techniques that we utilized for establishing the above convergence results . For example , the utilization of the local uniform metric subregularity is crucial in order to deal with the nonsmooth nonconvex term g in problem ( P ) , as it is far from obvious how to prove the previously used error bound condition if g is not null ( see Section 3.3 ) . Notations . We use 〈· , ·〉 and ‖ · ‖ to denote the Euclidean inner product and the Euclidean norm , respectively . For a matrix A , its minimum eigenvalue is denoted by λmin ( A ) . The spectral norm of a matrix A is denoted by ‖A‖ . Let S be a subset of Rd . We use projS ( z ) to denote the orthogonal projector onto S and dist ( z , S ) : = infx∈S ‖x − z‖ to denote the distance between x and S. When S = ∅ , we set dist ( z , S ) = +∞ . IS ( z ) = { 0 , z ∈ S +∞ , z /∈ S represents the indicator function of the set S. If S is a convex set , then the limiting normal cone to S is defined asNS ( z ) = { ξ : 〈ξ , ζ−z〉 ≤ 0 , ∀ζ ∈ S } .

Summary: This paper studies a particular type of nonconvex nonsmooth problem. The authors propose to use the Bertsekas' convexification procedure on the nonconvex problem, and then use randomized coordinate method to solve the corresponding augmented Lagrangian. The paper also proposes a so-called surrogate stationarity measure to establish the convergence rate for the proposed method.