Efficient High-Dimensional Data Representation Learning via Semi-Stochastic Block Coordinate Descent Methods

1 INTRODUCTION . In modern high-dimensional data analytics , where the variable dimension can be equal or even larger than the number of samples , sparse representation learning has become a mainstream method to explore the potential real model of the problem and provides statistically reliable results . It has been applied to many diverse domains such as high-dimensional statistics ( Bühlmann & Van De Geer , 2011 ) , signal processing ( Lai et al. , 2013 ) , and computer vision ( Wright et al. , 2008 ) . Many sparse learning algorithms such as ` 1-norm convex relaxation methods ( Negahban et al. , 2009 ; Van de Geer et al. , 2008 ) have been proposed in the past few decades . In order to get a smaller estimation error , ` 0-norm constrained algorithms are more prominent than ` 1-norm convex relaxation algorithms ( Zhang et al. , 2010 ) . In this paper , we mainly focus on the following sparsity-constrained optimization problem : min w F ( w ) : = 1 n n∑ i=1 fi ( w ) , s.t. , ‖w‖0 ≤ s , ( 1 ) where F ( w ) is a finite-sum convex and smooth function , each function fi ( w ) is associated with the i-th sample , ‖w‖0 is the number of nonzero entries in variable w , and s represents the sparsity level . The goal of sparse representation learning is to recover w∗ based on the given data . Such a formulation encapsulates plentiful important problems , including sparse graphical model learning ( Zhou et al. , 2019 ) , sparse linear/logistic regression ( Blumensath & Davies , 2009 ; Foucart , 2011 ; Pati et al. , 1993 ; Bahmani et al. , 2013 ) , and low-rank regression ( Rohde et al. , 2011 ) . However , Problem ( 1 ) is in general NP hard , which is caused by the non-convexity of the sparsity constraint . It makes us to obtain an approximate solution to Problem ( 1 ) . One of the most widely used methods for solving this problem is the hard thresholding based gradient descent method . In recent years , sparse representation learning is becoming more common for large-scale and highdimensional data . However , deterministic gradient descent hard thresholding algorithms such as fast gradient descent hard thresholding ( FG-HT ) ( Jain et al. , 2014 ; Yuan et al. , 2014 ) have to compute the gradients of all n component functions , which leads to a huge computing overhead . To address this issue , many stochastic hard thresholding methods have been proposed . For instance , Nguyen et al . ( 2017 ) proposed a stochastic gradient descent hard thresholding ( SG-HT ) algorithm , whose gradient and hard thresholding complexities are both O ( κs̃ log ( 1 ) ) . However , due to the stochastic sampling , SG-HT can only attain a sub-optimal estimation bound , as shown in Table 1 , which is inferior to those of deterministic gradient methods such as FG-HT . Another limitation of SG-HT is that it requires that the restricted condition number κs̃ should not be larger than 4/3 , which makes SG-HT hard for solving high-dimensional representation learning problems . Recently , many stochastic variance reduced methods ( e.g. , SAG ( Roux et al. , 2012 ) , SVRG ( Johnson & Zhang , 2013 ) ) and their accelerated variants such as ( Defazio , 2016 ; Allen-Zhu , 2018 ) have been proposed to accelerate stochastic gradient methods for convex optimization . All these methods enjoy low per-iteration complexities comparable with stochastic gradient descent ( SGD ) , and they also can attain improved convergence rates . By incorporating the variance reduction technique into sparsityconstrained optimization domain , Li et al . ( 2016b ) proposed a stochastic variance reduced gradient hard thresholding ( SVRG-HT ) algorithm , which can converge more quickly and obtain a smaller estimation error than SG-HT . Moreover , SVRG-HT allows an arbitrarily large condition number in its theoretical analysis similar to FG-HT ( Yuan et al. , 2014 ) . The gradient oracle and hard thresholding complexities of SVRG-HT are O ( ( n+κs̃ ) log ( 1 ) ) and O ( κs̃ log ( 1 ) ) , respectively . Nevertheless , it can not leverage the coordinate block to accelerate convergence . Chen & Gu ( 2016 ) proposed an accelerated stochastic block coordinate gradient descent hard thresholding ( ASBCDHT ) algorithm . The gradient oracle complexity of ASBCDHT is O ( ( n+ κs̃|B|k ) log ( 1 ) ) , which is superior to SVRG-HT . Moreover , ASBCDHT also has the hard thresholding complexity , O ( κs̃ log ( 1 ) ) . However , the hard thresholding complexity of ASBCDHT still scales linearly with κs̃ , which is usually expensive for real-world sparse learning problems . For large-scale and high-dimensional data , Li et al . ( 2016a ) also proposed the asynchronous parallel variant of SVRG-HT ( called ASVRG-HT ) by utilizing multicore architectures . Although it makes each processor to evaluate a stochastic gradient update on a global parameter stored in a shared memory in an asynchronous and lock-free mode , ASVRG-HT attains the similar gradient and hard thresholding complexities as its general version ( i.e. , SVRG-HT ) . Therefore , all the algorithms mentioned above have a linearly κs̃-dependent hard thresholding complexity . This motivates us to address the following key issue : Can we design such algorithms that have a κs̃-independent hard thresholding complexity and a lower gradient oracle complexity ? To answer the above problem , we propose an efficient Semi-stochastic Block Coordinate Descent Hard Thresholding Pursuit ( SBCD-HTP ) algorithm and its sparse Asynchronous variant ( ASBCDHTP ) . The oracle complexities and statistical estimation error of the proposed algorithms and other hard thresholding methods are summarized in Table 1 . We highlight several theoretical advantages of the proposed algorithms over the state-of-the-art methods as follows : • SBCD-HTP and ASBCD-HTP substantially improve the restricted condition on the sparsity level s , i.e. , s=Ω ( κŝs∗ ) , while most related algorithms such as SVRG-HT require s=Ω ( κ2s̃s ∗ ) . • The statistical estimation error of both our algorithms is better than those of FG-HT , SVRG-HT and ASBCDHT , as our algorithms have a smaller sparsity level s=Ω ( κŝs∗ ) and thus have a smaller cardinality , i.e. , ŝ= 2s+s∗ . This makes the restricted condition number κŝ of SBCD-HTP smaller than other algorithms . Moreover , the statistical error of our algorithms is also smaller than that of SG-HT ( i.e. , 1n ∑n i=1‖∇fi ( w∗ ) ‖ ) due to the large magnitude of ‖∇fi ( w∗ ) ‖ ( Zhou et al. , 2018b ) . • Both SBCD-HTP and ASBCD-HTP enjoy a κs̃-independent hard thresholding complexity , which is significantly better than the state-of-the-art hard thresholding algorithms . That is , the hard thresholding complexity of our algorithms is κs̃ times lower than that of FG-HT , SG-HT , SVRG-HT and ASBCDHT with s̃ = 2Ω ( κ2s̃s ∗ ) + s∗ . Since both our algorithms have a significantly lower hard thresholding complexity , they are more suitable for handling large-scale sparse representation learning problems , especially for high-dimensional data . • For sparsity-constrained problems , the gradient oracle complexities of both SBCD-HTP and ASBCD-HTP ( or S2BCD-HTP ) are much lower than those of the state-of-the-art hard thresholding algorithms , and that of S2BCD-HTP is similar to ASBCD-HTP under conditions on the delay . Actually , the gradient oracle complexity of S2BCD-HTP is better than SBCD-HTP when we deal with sparse data sets , since S2BCD-HTP only evaluates the common nonzero point of the random sample and the sampled block . Furthermore , their gradient oracle complexities are significantly lower than those of ASBCDHT and SVRG-HT , and much lower than that of FG-HT , since they require a smaller sparsity level s= Ω ( κŝs∗ ) and also have a smaller value of κŝ with ŝ= 2s+s∗ compared with the restricted condition number κs̃ used in other hard thresholding algorithms ( Zhou et al. , 2018b ) . 2 RELATED WORK . In this section , we briefly discuss the relevant research to our work , which beyonds the sparsityconstrained optimization domain . Traditional gradient descent is computational expensive at each iteration , and stochastic gradient descent has a low per-iteration complexity while obtaining a large variance in estimating . Thus , many stochastic variance reduced algorithms ( Defazio et al. , 2014 ; Roux et al. , 2012 ; Shalev-Shwartz & Zhang , 2013 ; Shang et al. , 2018 ; Johnson & Zhang , 2013 ) and their variants ( Schmidt et al. , 2017 ; Konečnỳ et al. , 2015 ; Xiao & Zhang , 2014 ; Hannah et al. , 2018 ) have been proposed . The stochastic variance reduced methods are very promising for many machine learning problems including sparse learning problems . In contrast to gradient descent methods , coordinate descent algorithms have received increasing attention due to their successful applications in high dimensional problems ( Breheny & Huang , 2011 ; Friedman et al. , 2007 ) . Among them , randomized block coordinate descent ( RBCD ) updates a block of coordinate with respect to the entire training instances . The per-iteration cost is significantly lower than gradient descent , while it still has a relationship with n component functions . Some stochastic block coordinate descent ( SBCD ) algorithms such as ( Dang & Lan , 2015 ; Xu & Yin , 2015 ; Konečnỳ et al. , 2017 ) were proposed . Such algorithms compute the stochastic gradient with respect to a random sample restricted to one randomized coordinate block . Therefore , these algorithms randomly sample a block of features and data instances at each iteration . However , they can only achieve a sublinear convergence rate ( Zhang & Gu , 2016 ) . Recently , some mini-batch randomized block coordinate descent algorithms such as ( Zhao et al. , 2014 ; Wang & Banerjee , 2014 ) were proposed to accelerate the convergence of stochastic block coordinate gradient descent . Our work studies over the above research by considering a sparsity-constrained optimization problem . The proposed algorithms enjoy a lower computational complexity in both gradient evaluation and hard thresholding computation while obtaining a linear convergence performance . Moreover , its sparse and asynchronous variants are also proposed to deal with sparse high-dimensional data . 3 PRELIMINARIES . Throughout this paper , we use w∗ to denote the optimal solution of Problem ( 1 ) , and its optimal sparsity level is s∗ with ‖w∗‖0≤ s∗ . ‖w‖ is the Euclidean norm for a vector w∈Rd and ‖w‖∞ is the largest absolute entry in w. The hard thresholding operation HT ( w , s ) preserves the s largest entries of w in magnitude for vector w and the rest entries are set to zero . We use supp ( w ) to denote the support set of w , i.e. , the indices of its nonzero entries . Given an index set I , we define IC as the complement set of I , and vI ∈ Rd , where [ vI ] j=vj if j∈I and [ vI ] j=0 if j /∈I . Given an integer Algorithm 1 Semi-stochastic Block Coordinate Descent Hard Thresholding Pursuit ( SBCD-HTP ) Input : Number of outer-loops R , number of inner-loops m , step size η , sparsity level s. Initialize : w̃0 . 1 : for r = 0 , 1 , . . . , R− 1 do 2 : w0 = w̃ = w̃r , ∇F ( w̃ ) = 1n ∑n i=1∇fi ( w̃ ) , G̃ = supp ( w̃ ) ; 3 : for t = 0 , 1 , . . . , m− 1 do 4 : Randomly sample a mini-batch B from [ n ] uniformly ; 5 : Randomly sample a block jt from [ k ] uniformly , and S = G̃ ∪ Gjt ; 6 : ∇Sg ( wt ) = 1|B| ∑ it∈B∇Sfit ( w t ) −∇Sfit ( w̃ ) +∇SF ( w̃ ) ; 7 : wt+1 = wt − η∇Sg ( wt ) ; 8 : end for 9 : w̃r+1 = HT ( wm , s ) ; 10 : end for Output : w̃R . n ≥ 1 , we define [ n ] = { 1 , · · · , n } . For a set B , we denote its cardinality by |B| . Moreover , we use the common notations of Ω ( · ) and O ( · ) to characterize the asymptotics of two real sequences . Throughout the analysis , we make two important assumptions on the objective function , which are commonly used in the analysis of hard thresholding algorithms ( Li et al. , 2016b ; Shen & Li , 2017 ; Chen & Gu , 2016 ; Gao & Huang , 2018 ) . In high-dimensional sparse learning , the per-iteration hard thresholding operation can be time-consuming or even more expensive than gradient computation . Thus , we also take the complexity of hard thresholding into our consideration . Assumption 1 ( Restricted Strong Convexity ) A differentiable function F ( · ) is restricted ρ−ŝ - strongly convex at sparsity level ŝ , if there exists a uniform constant ρ−ŝ > 0 such that for any w , w′ ∈ Rd with ‖w − w′‖0 ≤ ŝ , we have F ( w ) − F ( w′ ) − 〈∇F ( w′ ) , w − w′〉 ≥ ρ−ŝ 2 ‖w − w′‖2 . ( 2 ) Assumption 2 ( Restricted Strong Smoothness ) For any i ∈ [ n ] , the differentiable function fi ( · ) is restricted ρ+ŝ -strongly smooth at sparsity level ŝ , if there exists a uniform constant ρ + ŝ > 0 such that for any w , w′ ∈ Rd with ‖w − w′‖0 ≤ ŝ , we have fi ( w ) − fi ( w′ ) − 〈∇fi ( w′ ) , w − w′〉 ≤ ρ+ŝ 2 ‖w − w′‖2 . ( 3 ) Moreover , the restricted condition number is defined as : κŝ = ρ+ŝ /ρ − ŝ . Since the complexity of one hard thresholding operation is linear with d , which is similar to a computational cost of a stochastic gradient , we define the hard thresholding complexity as a metric for hard thresholding algorithms . Definition 1 ( Hard Thresholding Complexity ) In a hard thresholding operation , a vector w is fed into HT ( · , s ) and then the output HT ( w , s ) is obtained . Both gradient oracle and hard thresholding complexities can more comprehensively reflect the overall computational cost of first-order hard thresholding algorithms . Therefore , their per-iteration cost is dominated by both the gradient evaluation and hard thresholding operations . We also define the optimization error and statistical error to clearly understand our theoretical results . Definition 2 ( Optimization Error and Statistical Error ) The optimization error is the difference which will decrease with the increasing of iterations . A statistical error is the ( unknown ) difference between the retained value and the true value . This means that the optimization error will decrease to zero when the number of iteration is quite large . However , a statistical error will still work during the optimized process . Therefore , the larger the statistical error , the more difference to the true value .

Summary: This paper introduces a new optimization algorithm for convex functions with sparsity constraints, based on hard thresholding to enforce sparsity. It is argued that applying hard thresholding in every step is the computational bottleneck of previous works based on hard thresholding, and the paper proposes a variant that alleviates previous works’ convergence problems. This is achieved by having an inner and outer loop in which the hard thresholding is applied only in the outer loop, and in the inner loop block coordinate descent is used  (ie. applying updates to only a subset of the solution’s variables). The paper shows that the convergence bounds of the algorithms compares favourably to previous hard thresholding algorithms convergence bounds, and empirical results show the effectiveness of the algorithm compared to previous ones.