A Half-Space Stochastic Projected Gradient Method for Group Sparsity Regularization

1 INTRODUCTION . In many recent machine learning optimization tasks , researchers not only focus on finding solutions with small prediction/generalization error but also concentrate on improving the interpretation of model by filtering out redundant parameters and achieving slimmer model architectures . One technique to achieve the above goal is by augmenting the sparsity-inducing regularization terms to the raw objective functions to generate sparse solutions ( including numerous zero elements ) . The popular ` 1-regularization promotes the sparsity of solutions by element-wise penalizing the optimization variables . However , in many practical applications , there exist additional constraints on variables such that the zero coefficients are often not randomly distributed but tend to be clustered into varying more sophisticated sparsity structures , e.g. , disjoint and overlapping groups and hierarchy ( Yuan & Lin , 2006 ; Huang et al. , 2010 ; 2009 ) . As the most important and natural form of structured sparsity , the disjoint group-sparsity regularization , which assumes the pre-specified disjoint blocks of variables are selected ( non-zero variables ) or ignored ( zero variables ) simultaneously ( Bach et al. , 2012 ) , serves as a momentous role in general structured sparsity learning tasks since other instances such as overlapping group and hierarchical sparsity are typically solved by converting into the equivalent disjoint group versions via introducing latent variables ( Bach et al. , 2012 ) , and has found numerous applications in computer vision ( Elhamifar et al. , 2012 ) , signal processing ( Chen & Selesnick , 2014 ) , medical imaging ( Liu et al. , 2018 ) , and deep learning ( Scardapane et al. , 2017 ) , especially on the model compression of deep neural networks , where the group sparsity1 is leveraged to remove redundant entire hidden structures directly . Problem Setting . We study the disjoint group sparsity regularization problem which can be typically formulated as the mixed ` 1/ ` p-regularization problem , and pay special attention to the most popular and widely used instance p as 2 ( Bach et al. , 2012 ; Halabi et al. , 2018 ) , minimize x∈Rn { Ψ ( x ) def = f ( x ) + λΩ ( x ) = 1 N N∑ i=1 fi ( x ) + λ ∑ g∈G ‖ [ x ] g‖ } , ( 1 ) where λ > 0 is a weighting factor , ‖·‖ denotes ` 2-norm , f ( x ) is the average of numerous N continuously differentiable instance functions fi : Rn → R , such as the loss functions measuring the deviation from the observations in various data fitting problems , Ω ( x ) is the so-called mixed ` 1/ ` 2 1Group sparsity is defined as # of zero groups , where a zero group means all its variables are exact zeros . norm , G is a prescribed fixed partition of index set I = { 1 , 2 , · · · , n } , wherein each component g ∈ G indexes a group of variables upon the perspective of applications . Theoretically , a larger λ typically results in a higher group sparsity while sacrifices more on the bias of model estimation , hence λ needs to be carefully fine-tuned to achieve both low f and high group-sparse solutions . Literature Review . Problem ( 1 ) has been well studied in deterministic optimization with various algorithms that are capable of returning solutions with both low objective value and high group sparsity under proper λ ( Yuan & Lin , 2006 ; Roth & Fischer , 2008 ; Huang et al. , 2011 ; Ndiaye et al. , 2017 ) . Proximal methods are classical approaches to solve the structured non-smooth optimization ( 1 ) , including the popular proximal gradient method ( Prox-FG ) which only uses the first-order derivative information . When N is huge , stochastic methods become ubiquitous to operate on a small subset to avoid the costly evaluation over all instances in deterministic methods for large-scale problems . Proximal stochastic gradient method ( Prox-SG ) ( Duchi & Singer , 2009 ) is the natural stochastic extension of Prox-FG . Regularized dual-averaging method ( RDA ) ( Xiao , 2010 ; Yang et al. , 2010 ) is proposed by extending the dual averaging scheme in ( Nesterov , 2009 ) . To improve the convergence rate , there exists a set of incremental gradient methods inspired by SAG ( Roux et al. , 2012 ) to utilizes the average of accumulated past gradients . For example , proximal stochastic variance-reduced gradient method ( Prox-SVRG ) ( Xiao & Zhang , 2014 ) and proximal spider ( Prox-Spider ) ( Zhang & Xiao , 2019 ) are developed to adopt multi-stage schemes based on the well-known variance reduction technique SVRG proposed in ( Johnson & Zhang , 2013 ) and Spider developed in ( Fang et al. , 2018 ) respectively . SAGA ( Defazio et al. , 2014 ) stands as the midpoint between SAG and Prox-SVRG . Compared to deterministic methods , the studies of mixed ` 1/ ` 2-regularization ( 1 ) in stochastic field become somewhat rare and limited . Prox-SG , RDA , Prox-SVRG , Prox-Spider and SAGA are valuable state-of-the-art stochastic algorithms for solving problem ( 1 ) but with apparent weakness . Particularly , these existing stochastic algorithms typically meet difficulties to achieve both decent convergence and effective group sparsity identification simultaneously ( e.g. , small function values but merely dense solutions ) , because of the randomness and the limited sparsity-promotion mechanisms . In depth , Prox-SG , RDA , Prox-SVRG , Prox-Spider and SAGA derive from proximal gradient method to utilize the proximal operator to produce group of zero variables . Such operator is generic to extensive non-smooth problems , consequently perhaps not sufficiently insightful if the target problems possess certain properties , e.g. , the group sparsity structure as problem ( 1 ) . In fact , in convex setting , the proximal operator suffers from variance of gradient estimate ; and in non-convex setting , especially deep learning , the discreet step size ( learning rate ) further deteriorates its effectiveness on the group sparsity promotion , as will show in Section 2 that the projection region vanishes rapidly except RDA . RDA has superiority on finding manifold structure to others ( Lee & Wright , 2012 ) , but inferiority on the objective convergence . Besides , the variance reduction techniques are typically required to measure over a huge mini-batch data points in both theory and practice which is probably prohibitive for large-scale problems , and have been observed as sometimes noneffective for deep learning applications ( Defazio & Bottou , 2019 ) . On the other hand , to introduce sparsity , there exist heuristic weight pruning methods ( Li et al. , 2016 ; Luo et al. , 2017 ) , whereas they commonly do not equip with theoretical guarantee , so that easily diverge and hurt generalization accuracy . Our Contributions . Half-Space Stochastic Projected Gradient ( HSPG ) method overcomes the limitations of the existing stochastic algorithms on the group sparsity identification , while maintains comparable convergence characteristics . While the main-stream works on ( group ) sparsity have focused on using proximal operators of regularization , our method is unique and fresh in enforcing group sparsity more effectively by leveraging half-space structure and is well supported by the theoretical analysis and empirical evaluations . We now summarize our contributions as follows . • Algorithmic Design : We propose the HSPG to solve the disjoint group sparsity regularized problem as ( 1 ) . Initialized with a Prox-SG Step for seeking a close-enough but perhaps dense solution estimate , the algorithmic framework relies on a novel Half-Space Step to exploit group sparse patterns . We delicately design the Half-Space Step with the following main features : ( i ) it utilizes previous iterate as the normal direction to construct a reduced space consisting of a set of halfspaces and the origin ; ( ii ) a new group projection operator maps groups of variables onto zero if they fall out of the constructed reduced space to identify group sparsity considerably more effectively than the proximal operator ; and ( iii ) with proper step size , the Half-Space Step enjoys the sufficient decrease property , and achieves progress to optimum in both theory and practice . • Theoretical Guarantee : We provide the convergence guarantees of HSPG . Moreover , we prove HSPG has looser requirements to identify the sparsity pattern than Prox-SG , revealing its superiority on the group sparsity exploration . Particularly , for the sparsity pattern identification , the required distance to the optimal solution x∗ of HSPG is better than the distance required by Prox-SG . • Numerical Experiments : Experimentally , HSPG outperforms the state-of-the-art methods in the aspect of the group sparsity exploration , and achieves competitive objective value convergence and runtime in both convex and non-convex problems . In the popular deep learning tasks , HSPG usually computes the solutions with multiple times higher group sparsity and similar generalization performance on unseen testing data than those generated by the competitors , which may be further used to construct smaller and more efficient network architectures . 2 THE HSPG METHOD . We state the Half-Space Stochastic Projected Gradient ( HSPG ) method in Algorithm 1 . In general , it contains two stages : Initialization Stage and Group-Sparsity Stage . The first Initialization Stage employs Prox-SG Step ( Algorithm 2 ) to search for a close-enough but usually non-sparse solution estimate . Then the second and fundamental stage proceeds Half-Space Step ( Algorithm 3 ) started with the non-sparse solution estimate to effectively exploit the group sparsity within a sequence of reduced spaces , and converges to the group-sparse solutions with theoretical convergence property . Algorithm 1 Outline of HSPG for solving ( 1 ) . 1 : Input : x0 ∈ Rn , α0 ∈ ( 0 , 1 ) , ∈ [ 0 , 1 ) , and NP ∈ Z+ . 2 : for k = 0 , 1 , 2 , . . . do 3 : if k < NP then 4 : Compute xk+1 ← Prox-SG ( xk , αk ) by Algorithm 2 . 5 : else 6 : Compute xk+1 ← Half-Space ( xk , αk , ) by Algorithm 3 . 7 : Update αk+1 . Algorithm 2 Prox-SG Step . 1 : Input : Current iterate xk , and step size αk . 2 : Compute the stochastic gradient of f on mini-batch Bk ∇fBk ( xk ) ← 1 |Bk| ∑ i∈Bk ∇fi ( xk ) . ( 2 ) 3 : Return xk+1 ← ProxαkλΩ ( · ) ( xk − αk∇fBk ( xk ) ) . Initialization Stage . The Initialization Stage performs the vanilla proximal stochastic gradient method ( Prox-SG , Algorithm 2 ) to approach the solution of ( 1 ) . At kth iteration , a mini-batch Bk is sampled to generate an unbiased estimator of the full gradient of f ( line 2 , Algorithm 2 ) to compute a trial iterate x̂k+1 : = xk − αk∇fBk ( xk ) , where αk is the step size , and fBk is the average of the instance functions fi cross Bk . The next iterate xk+1 is then updated based on the proximal mapping xk+1 = ProxαkλΩ ( · ) ( x̂k+1 ) = arg min x∈Rn 1 2αk ‖x− x̂k+1‖2 + λΩ ( x ) , ( 3 ) where the regularization term Ω ( x ) is defined in ( 1 ) . Notice that the above subproblem ( 3 ) has a closed-form solution , where for each g ∈ G , we have [ xk+1 ] g = max { 0 , 1− αkλ/ ‖ [ x̂k+1 ] g‖ } · [ x̂k+1 ] g. ( 4 ) In HSPG , the Initialization Stage proceeds Prox-SG Step NP times as a localization mechanism to seek an estimation which is close enough to a solution of problem ( 1 ) , where NP : = min { k : k ∈ Z+ , ‖xk − x∗‖ ≤ R/2 } associated with a positive constant R related to the optima , see ( 23 ) in Appendix C. In practice , although the close-enough requirement is perhaps hard to be verified , we empirically suggest to keep running the Prox-SG Step until observing some stage-switch signal by testing on the stationarity of objective values , norm of ( sub ) gradient or validation accuracy similarly to ( Zhang et al. , 2020 ) . However , the Initialization Stage alone is insufficient to exploit the group sparsity structure , i.e. , the computed solution estimate is typically dense , due to the randomness and the moderate truncation mechanism of proximal operator constrained in its projection region , i.e. , the trial iterate [ x̂k+1 ] g is projected to zero only if it falls into an ` 2-ball centered at the origin with radius αkλ by ( 4 ) . Our remedy is to incorporate it with the following Half-Space Step , which exhibits an effective sparsity promotion mechanism while still remains the convergent property . Algorithm 3 Half-Space Step 1 : Input : Current iterate xk , step size αk , and . 2 : Compute the stochastic gradient of Ψ on I 6=0 ( xk ) by mini-batch Bk [ ∇ΨBk ( xk ) ] I 6=0 ( xk ) ← 1 |Bk| ∑ i∈Bk [ ∇Ψi ( xk ) ] I 6=0 ( xk ) ( 5 ) 3 : Compute [ x̃k+1 ] I 6=0 ( xk ) ← [ xk − αk∇ΨBk ( xk ) ] I 6=0 ( xk ) and [ x̃k+1 ] I0 ( xk ) ← 0 . 4 : for each group g in I 6=0 ( xk ) do 5 : if [ x̃k+1 ] > g [ xk ] g < ‖ [ xk ] g‖ 2 then 6 : [ x̃k+1 ] g ← 0 . 7 : Return xk+1 ← x̃k+1 . Group-Sparsity Stage . The Group-Sparsity Stage is designed to effectively determine the groups of zero variables and capitalize convergence characteristic , which is in sharp contrast to other heuristic aggressive weight pruning methods but typically lacking theoretical guarantee ( Li et al. , 2016 ; Luo et al. , 2017 ) . The underlying intuition of its atomic Half-Space Step ( Algorithm 3 ) is to project [ xk ] g to zero only if − [ xk ] g serves as a descent step to Ψ ( xk ) , i.e. , − [ xk ] > g [ ∇Ψ ( xk ) ) ] g < 0 , hence updating [ xk+1 ] g ← [ xk ] g − [ xk ] g = 0 still results in some progress to the optimality . Before introducing that , we first define the following index sets for any x ∈ Rn : I0 ( x ) : = { g : g ∈ G , [ x ] g = 0 } and I 6=0 ( x ) : = { g : g ∈ G , [ x ] g 6= 0 } , ( 6 ) where I0 ( x ) represents the indices of groups of zero variables at x , and I 6=0 ( x ) indexes the groups of nonzero variables at x . To proceed , we further define an artificial set that x lies in : S ( x ) : = { z ∈ Rn : [ z ] g = 0 if g ∈ I0 ( x ) , and [ z ] > g [ x ] g ≥ ‖ [ x ] g‖ 2 if g ∈ I 6=0 ( x ) } ⋃ { 0 } , ( 7 ) which consists of half-spaces and the origin . Here the parameter > 0 controls the grey region presented in Figure 1b , and the exact way to set will be discussed in Section 4 and Appendix . Hence , x inhabits S ( xk ) , i.e. , x ∈ S ( xk ) , only if : ( i ) [ x ] g lies in the upper half-space for all g ∈ I 6=0 ( xk ) for some prescribed ∈ [ 0 , 1 ) as shown in Figure 1a ; and ( ii ) [ x ] g equals to zero for all g ∈ I0 ( xk ) . The fundamental assumption for Half-Space Step to success is that : the Initialization Stage has produced a ( possibly non-sparse ) solution estimate xk nearby a group sparse solution x∗ of problem ( 1 ) , i.e. , the optimal distance ‖xk − x∗‖ is sufficiently small . As seen in Appendix , it further indicates that the group sparse optimal solution x∗ inhabits Sk : = S ( xk ) , which implies that Sk has already covered the group-support of x∗ , i.e. , I 6=0 ( x∗ ) ⊆ I 6=0 ( xk ) . Our goal now becomes minimizing Ψ ( x ) over Sk to identify the remaining groups of zero variables , i.e. , I0 ( x∗ ) /I0 ( xk ) , which is formulated as the following smooth optimization problem : xk+1 = arg min x∈Sk Ψ ( x ) = f ( x ) + λΩ ( x ) . ( 8 ) By the definition of Sk , [ x ] I0 ( xk ) ≡ 0 are constrained as fixed during Algorithm 3 proceeding , and only the entries in I 6=0 ( xk ) are allowed to move . Hence Ψ ( x ) is smooth on Sk , and ( 8 ) is a reduced space optimization problem . A standard way to solve problem ( 8 ) would be the stochastic gradient descent equipped with Euclidean projection ( Nocedal & Wright , 2006 ) . However , such a projected method rarely produces zero ( group ) variables as the dense x̂E illustrated in Figure 1a . To address it , we introduce a novel projection operator to effectively conduct group projection as follows . As stated in Algorithm 3 , we first approximate the gradient of Ψ on the free variables in I 6=0 ( xk ) by [ ∇ΨBk ( xk ) ] I 6=0 ( xk ) ( line 2 , Algorithm 3 ) , then employ SGD to compute a trial point x̃k+1 ( line 3 , Algorithm 3 ) which is passed into a new projection operator ProjSk ( · ) defined as [ ProjSk ( z ) ] g : = { [ z ] g if [ z ] > g [ xk ] g ≥ ‖ [ xk ] g‖ 2 , 0 otherwise . ( 9 ) The above projector of form ( 9 ) is not the standard Euclidean projection operator in most cases2 , but still satisfies the following two advantages : ( i ) the actual search direction dk : = ( ProjSk ( x̃k+1 ) − xk ) /αk performs as a descent direction to ΨBk ( xk ) : = fBk ( xk ) + λΩ ( xk ) , i.e. , [ dk ] > g [ ∇ΨBk ( xk ) ) ] g < 0 as θ < 90◦ in Figure 1a , then the progress to the optimum is made via the sufficient decrease property as drawn in Lemma 1 ; and ( ii ) effectively project groups of variables to zero simultaneously if the inner product of corresponding entries is sufficiently small . In contrast , the Euclidean projection operator is far away effective to promote group sparsity , as the Euclidean projected point x̂E 6= 0 versus xk+1 = ProjSk ( x̃k+1 ) = 0 shown in Figure 1a . Lemma 1 . Algorithm 3 yields the next iteratexk+1 as ProjSk ( xk−αk∇ΨBk ( xk ) ) , then the search direction dk : = ( xk+1−xk ) /αk is a descent direction for ΨBk ( xk ) , i.e. , d > k∇ΨBk ( xk ) < 0 . Moreover , letting L be the Lipschitz constant for∇ΨBk on the feasible domain , and Ĝk : = I 6=0 ( xk ) ⋂ I0 ( xk+1 ) and G̃k : = I 6=0 ( xk ) ⋂ I 6=0 ( xk+1 ) be the sets of groups which projects or not onto zero , we have ΨBk ( xk+1 ) ≤ΨBk ( xk ) − ( αk − α2kL 2 ) ∑ g∈G̃k ‖ [ ∇ΨBk ( xk ) ] g‖ 2 − ( 1− αk − L 2 ) ∑ g∈Ĝk ‖ [ xk ] g‖2 . ( 10 ) We then intuitively illustrate the strength of HSPG on group sparsity exploration . In fact , the halfspace projection ( 9 ) is a more effective sparsity promotion mechanism compared to the existing methods . Particularly , it benefits from a much larger projection region to map a reference point x̂k+1 : = xk − αk∇fBk ( xk ) or its variants to zero . As the 2D case described in Figure 1b , the projection regions of Prox-SG , Prox-SVRG , Prox-Spider and SAGA are ` 2-balls with radius as αkλ . In stochastic learning , especially deep learning tasks , the step size αk is usually selected around 10−3 to 10−4 or even smaller for convergence . Together with the common setting of λ 1 , their projection regions would vanish rapidly , resulting in the difficulties to produce group sparsity . As a sharp contrast , even though αkλ is near zero , the projection region of HSPG { x : x > k x < ( αkλ+ ‖xk‖ ) ‖xk‖ } ( seen in Appendix ) is still an open half-space which contains those ` 2 balls as well as RDA ’ s if is large enough . Moreover , the positive control parameter adjusts the level of aggressiveness of group sparsity promotion ( 9 ) , i.e. , the larger the more aggressive , and meanwhile maintains the progress to the optimality by Lemma 1 . In practice , proper fine tuning is sometimes required to achieve both group sparsity enhancement and sufficient decrease on objective value as will see in Section 4 . Intuition of Two-Stage Method : To end this section , we discuss the advantage of designing such two stage schema rather than an adaptive switch back and forth between the Prox-SG Step and Half-Space Step based on some evaluation switching criteria , as many multi-step deterministic optimization algorithms ( Chen et al. , 2017 ) . In fact , we numerically observed that switching back to the Prox-SG Step consistently deteriorate the progress of group sparsity exploration by Half-Space Step while without obvious gain on convergence . Such regression on group sparsity by the Prox-SG Step 2Unless Ω ( x ) is ‖x‖1 where each g ∈ G is singleton , then Sk becomes an orthant face ( Chen et al. , 2020 ) . is less attractive in realistic applications , e.g. , model compression , where people usually possess heavy models of high generalization accuracy ahead and want to filter out the redundancy effectively . Therefore , in term of the ease of application , we end at organizing Prox-SG Step and Half-Space Step as such a two-stage schema , controlled by a switching hypermeter NP . In theory , we require NP sufficiently large to let the initial iterate of Half-Space Step be close enough to the local minimizer as shown in Section 3 . In practice , HSPG is sensitive to the choice of NP at early iterations , i.e. , switching to Half-Space Step too early may result accuracy loss . But such sensitivity vanishes rapidly if switching to Half-Space Step after some acceptable evaluation switching criteria .

Summary: The paper studies how to solve a class of group sparsity regularized minimization problems. In particular, a half-space stochastic projected gradient (HSPG) method is proposed, which is based on the Prox-SG and a new half-space step that promotes group sparsity. This step is to decompose the feasible space and then perform group projection. Convergence analysis is provided, together with the theoretical discussion that HSPG has looser requirements to identify the sparsity patter than Prox-SG. Numerical experiments on the DCNNs based image classification shows the proposed method achieves the state-of-the-art performance in terms of accuracy. The work looks interesting with wide applications, especially in deep neural networks. However, the novelty is incremental and limited.