Randomized Primal-Dual Coordinate Method for Large-scale Linearly Constrained Nonsmooth Nonconvex Optimization

The large-scale linearly constrained nonsmooth nonconvex optimization finds wide applications in machine learning , including non-PSD Kernel SVM , linearly constrained Lasso with nonsmooth nonconvex penalty , etc . To tackle this class of optimization problems , we propose an efficient algorithm called Nonconvex Randomized Primal-Dual Coordinate ( N-RPDC ) method . At each iteration , this method only randomly selects a block of primal variables to update rather than updating all the variables , which is suitable for large-scale problems . We provide two types of convergence results for N-RPDC . We first show that any cluster point of the sequence of iterates generated by N-RPDC is almost surely ( i.e. , with probability 1 ) a stationary point . In addition , we also provide an almost sure asymptotic convergence rate of O ( 1/ √ k ) . Next , we establish the expected O ( ε−2 ) iteration complexity of N-RPDC in order to drive a natural stationarity measure below ε in expectation . The fundamental aspect to establishing the aforementioned convergence results is a surrogate stationarity measure we discovered for analyzing N-RPDC . Finally , we conduct a set of experiments to show the efficacy of NRPDC . 1 INTRODUCTION . Many large scale problems arising in machine learning amounts to solving the following linearly constrained nonsmooth nonconvex optimization problem : min x∈X F ( x ) = f ( x ) + g ( x ) s.t Ax− b = 0 , ( P ) where A ∈ Rn×d and b ∈ Rn . Throughout this paper , we impose the following assumptions on problem ( P ) : the function f : Rd → R is possibly nonconvex and continuously differentiable with its gradient ∇f being Lf -Lipschitz continuous , X is a convex and compact set of Rd , i.e. , there exits a positive number M such that M = maxx , x′∈X ‖x− x′‖ , and g : Rd → R is nonsmooth and nonconvex . More precisely , g is assumed to be lower semicontinuous ( l.s.c ) ρg-weakly convex and has bounded subgradients over X , i.e. , there exists Lg > 0 such that ‖s‖ ≤ Lg , ∀s ∈ ∂g ( x ) , x ∈ X , where ∂g is the subdifferential of g ( see ( 4 ) for definition ) . Let F ∗ be the optimal value of ( P ) . Recall that a function g is said to be ρg-weakly convex if g ( · ) + ρg2 ‖ · ‖ 2 is convex for some constant ρg ≥ 0 ( Vial , 1983 ) . It is worth mentioning that a wide class of nonsmooth nonconvex functions belong to the weakly convex class ; see , e.g. , ( Vial , 1983 ; Davis & Drusvyatskiy , 2019 ) for more discussions on weak convexity . We further assume g ( x ) = ∑N i=1 gi ( xi ) is block separable with respect to the space decomposition X = X1 × X2 × · · · × XN , xi ∈ Xi ⊂ Rdi , and N∑ i=1 di = d. ( 1 ) Let A = ( A1 , A2 , . . . , AN ) ∈ Rn×d be the corresponding partition of A , where Ai ∈ Rn×di . When N = d , each Xi ⊂ R corresponds to a box constraint on the coordinate xi for i = 1 , . . . , d. Motivations . Our motivation for studying large scale problem ( P ) stems from the fact that this problem class finds wide applications in machine learning . To be more specific , we will present several practical examples below that give rise to ( P ) . Application 1 : Non-PSD kernel support vector machine . Support vector machine ( SVM ) is a widely utilized technique for supervised learning ( Boser et al. , 1992 ; Cortes & Vapnik , 1995 ) . In order to improve the interpretability and enhance the robustness , some non-PSD kernels are used in the kernalized SVM method , such as the sigmoid kernel ( Lin & Lin , 2003 ) , the jittering kernel ( DeCoste & Schölkopf , 2002 ) , the tangent distance kernel ( Haasdonk & Keysers , 2002 , August ) , to name a few . Formally , the kernalized SVM problem can be written as min x∈ [ 0 , c ] d 1 2x > Qx− 1 > d x s.t . y > x = 0 , ( 2 ) where Q is a d× d non-PSD matrix , c ∈ R is the upper bound of all variables , y ∈ { −1 , 1 } d is the vector of labels , and 1d is an d-dimensional vector of all 1s . Application 2 : Linearly constrained Lasso with nonsmooth nonconvex penalty . The Lasso is one of the most popular methods for variable selection . The standard Lasso uses ` 1-norm to select important variables . Some works propose to utilize nonsmooth nonconvex regularizers to further improve the performance of the Lasso ; see , e.g. , ( Breheny & Huang , 2011 ; 2015 ; Rakotomamonjy et al. , 2019 ) . If additional prior information is available , the works ( Deng et al. , 2020 , May ; Gaines et al. , 2018 ; James et al. , 2013 ; Won et al. , 2019 ) propose the constrained Lasso model . The linearly constrained LASSO with nonsmooth nonconvex penalty can be formulated as follows : min x∈X 1 2‖Ax− b‖ 2 + d∑ i=1 φ ( xi ) s.t . Bx− c = 0 ( 3 ) where x = ( x1 , ... , xd ) > , X ⊂ Rd , A ∈ Rn×d is the design matrix , b ∈ Rn is the response vector , B ∈ Rm×d and c ∈ Rm are given constraints , and φ is a nonsmooth nonconvex regularizer such as MCP ( Zhang et al. , 2010 ) or SCAD ( Fan & Li , 2001 ) penalty . There are plenty of other applications that give rise to ( P ) , e.g. , the robust M-estimators , distributed learning , etc . Due to the limitation of space , we will not expand them in details here . Related works . Linearly constrained nonconvex optimization . Perhaps the most widely utilized class of algorithms for solving constrained optimization problems are the Primal-dual methods . Previous works on single-loop primal-dual methods mainly consider linearly contained convex optimization problems . The work ( Boţ & Nguyen , 2020 ) considers a nonconvex instance with a special linear constraint Ax− z = 0 , where both x and z are decision variables and the matrix A is assumed to be surjective . Under these assumptions , convergence results were established for a proximal alternating direction method of multipliers ( ADMM ) . The recent works ( Zhang & Luo , 2020a ; b ) provide convergence results of ADMM using gradient-based updates for linearly constrained smooth nonconvex optimization problems with a general linear constraint Ax = b , i.e. , problem ( P ) with g ≡ 0 . The main insights of their results are a construction of an proximally regularized auxiliary problem that dates back to ( Bertsekas , 1979 ) and a properly constructed Lyapunov function . Then , they established descent property on the Lyapunov function based on the dual error bound condition ( see also ( Hong & Luo , 2017 ) ) , which leads to iteration complexity results . The method introduced in ( Zhu et al. , 2020 ) can be used to solve problem ( P ) with specific linear constraints . This method is based on the auxiliary problem principle of augmented Lagrangian ( APP-AL ) method ( Cohen & Zhu , 1984 ; Zhao & Zhu , 2019 ; 2021 ) , which can be seen as a forward-backward splitting method and applies a Jacobian updating strategy.1 However , both ADMM-type and APP-AL-type methods compute the full gradient of primal variables and update all primal variables at each iteration . Therefore , the computation complexity of one iteration of these two types of methods are expensive for large-scale problems ( See Remark 2.1 ) . Randomized coordinate methods for unconstrained optimization . In the past decade , big data applications are ubiquitous in machine learning . The formulated optimization problems often involve very large datasets , and hence computing the function value or the gradient can be very expensive . These 1ADMM applies a Gauss-Seidel like minimization strategy , which is related to the Douglas-Rachford splitting method . observations motivate the study of the randomized coordinate methods . The researches on this type of methods can be traced back to ( Nesterov , 2012 ; 2014 ) . In ( Nesterov , 2012 ) , Nesterov studied the iteration complexity of the randomized coordinate gradient ( RCG ) descent method for smooth convex optimization . Later , in ( Nesterov , 2014 ) , the same author analyzed the randomized coordinate subgradient method for a set of piece-wise linear nonsmooth convex optimization problems . The works ( Richtárik & Takáč , 2014 ) and ( Lu & Xiao , 2015 ) extends Nesterov ’ s results to convex additive composite optimization . Iteration complexity and linear convergence of RCG were studied in ( Patrascu & Necoara , 2015 ) and ( Karimi et al. , 2016 ) for nonsmooth nonconvex optimization and smooth nonconvex optimization , respectively . Moreover , by employing an asynchronous point to evaluate the gradient in each iteration , The works ( Liu & Wright , 2015 ) and ( Liu et al. , 2014 ) establish the iteration complexity results of asynchronous RCG for convex smooth optimization and additive composite optimization , respectively . A Few results on randomized coordinate methods for linearly constrained optimization . Though the randomized coordinate-type methods for unconstrained optimization are extensively studied , there are only a few results for this type of methods concerning coupled linearly constrained optimization problems . The works ( Gao et al. , 2019 ; Wang et al. , 2014 ) study the randomized primal-dual coordinate ( RPDC ) method and provides iteration complexity results for linearly constrained convex optimization . Better complexity results of RPDC were obtained for linearly constrained strongly convex optimization ( Xu & Zhang , 2018 ) . The recent works ( Zhu & Zhao , 2020 ; Latafat et al. , 2019 ; Fercoq & Bianchi , 2019 ; Alacaoglu et al. , 2020 ) provide almost sure ( i.e. , with probability 1 ) asymptotic convergence results of RPDC-type methods for linearly constrained convex optimization . It is worth emphasizing that the works ( Latafat et al. , 2019 ; Fercoq & Bianchi , 2019 ; Alacaoglu et al. , 2020 ) establish sequential convergence results ( i.e. , the convergence of the whole sequence of iterates ) . However , the above mentioned works only considers either ADMM-type and APP-AL-type methods for linearly constrained smooth nonconvex optimization problem or the randomized coordinate methods for linearly constrained convex optimization . To the best of our knowledge , no previous result concerns convergence and iterate complexity of the randomized coordinate methods for linearly constrained nonsmooth nonconvex optimization , which will be the main focus of this paper . Main contributions . In this paper , we aim to solve the large-scale linearly constrained nonsmooth nonconvex optimization problem ( P ) . To tackle it , we propose the Nonconvex Randomized PrimalDual Coordinate ( N-RPDC ) method based on an auxiliary problem ( see Section 2 ) . Relying on the mild local uniform metric subregularity property ( see Section 3.3 ) , we provide two types of convergence results for N-RPDC ( see Section 4 ) . We first show that any cluster point of the sequence of iterates generated by N-RPDC is almost surely ( i.e. , with probability 1 ) a stationary point of ( P ) . In addition , we also provide an almost sure asymptotic convergence rate of O ( 1/ √ k ) , where k represents iteration number . Next , we establish the expected iteration complexity of N-RPDC . Namely , N-RPDC needs at most O ( ε−2 ) number of iterations in order to drive a surrogate stationarity measure below ε in expectation . The fundamental aspect to our convergence analysis is a surrogate stationarity measure we discovered for analyzing N-RPDC ( see Section 3.1 ) , which is one of the main contributions of this work . The standard stationarity measure ( i.e. , checking KKT conditions ) is not directly applicable here due to the stochastic nature of N-RPDC . Instead , we define the notion of reference point and use its distance to the iterate generated by N-RPDC as a surrogate stationarity measure . Then , such a surrogate stationarity measure is clarified by showing that it is an upper bound of the standard stationarity measure at the reference point up to a numerical constant ( see Proposition 3.1 and Remark 3.1 ) . There are also some other interesting techniques that we utilized for establishing the above convergence results . For example , the utilization of the local uniform metric subregularity is crucial in order to deal with the nonsmooth nonconvex term g in problem ( P ) , as it is far from obvious how to prove the previously used error bound condition if g is not null ( see Section 3.3 ) . Notations . We use 〈· , ·〉 and ‖ · ‖ to denote the Euclidean inner product and the Euclidean norm , respectively . For a matrix A , its minimum eigenvalue is denoted by λmin ( A ) . The spectral norm of a matrix A is denoted by ‖A‖ . Let S be a subset of Rd . We use projS ( z ) to denote the orthogonal projector onto S and dist ( z , S ) : = infx∈S ‖x − z‖ to denote the distance between x and S. When S = ∅ , we set dist ( z , S ) = +∞ . IS ( z ) = { 0 , z ∈ S +∞ , z /∈ S represents the indicator function of the set S. If S is a convex set , then the limiting normal cone to S is defined asNS ( z ) = { ξ : 〈ξ , ζ−z〉 ≤ 0 , ∀ζ ∈ S } .

Summary: The paper proposes a randomized primal-dual method for solving the nonconvex nonsmooth optimization problem with linear constraints. The main analysis relies on the idea of using auxiliary problems and the Lyapunov function proposed in  (Zhang & Luo, 2020b).  The paper shows that any cluster point of the generated sequence is almost surely a stationary point of the nonconvex nonsmooth problem and also provides an almost sure asymptotic convergence rate together with the expected iteration complexity of the proposed method.