Adaptive Optimizers with Sparse Group Lasso

1 INTRODUCTION . With the development of deep learning , deep neural network ( DNN ) models have been widely used in various machine learning scenarios such as search , recommendation and advertisement , and achieved significant improvements . In the last decades , different kinds of optimization methods based on the variations of stochastic gradient descent ( SGD ) have been invented for training DNN models . However , most optimizers can not directly produce sparsity which has been proven effective and efficient for saving computational resource and improving model performance especially in the scenarios of very high-dimensional data . Meanwhile , the simple rounding approach is very unreliable due to the inherent low accuracy of these optimizers . In this paper , we develop a new class of optimization methods , that adds the regularizers especially sparse group lasso to prevalent adaptive optimizers , and retains the characteristics of the respective optimizers . Compared with the original optimizers with the post-processing procedure which use the magnitude pruning method , the performance of the models can be significantly improved on the same sparsity level . Furthermore , in comparison to the cases without magnitude pruning , the new optimizers can achieve extremely high sparsity with significantly better or highly competitive performance . In this section , we describe the two types of optimization methods , and explain the motivation of our work . 1.1 ADAPTIVE OPTIMIZATION METHODS . Due to the simplicity and effectiveness , adaptive optimization methods ( Robbins & Monro , 1951 ; Polyak , 1964 ; Duchi et al. , 2011 ; Zeiler , 2012 ; Kingma & Ba , 2015 ; Reddi et al. , 2018 ; Yao et al. , 2020 ) have become the de-facto algorithms used in deep learning . There are multiple variants , but they can be represented using the general update formula ( Reddi et al. , 2018 ) : xt+1 = xt − αtmt/ √ Vt , ( 1 ) where αt is the step size , mt is the first moment term which is the weighted average of gradient gt and Vt is the so called second moment term that adjusts updated velocity of variable xt in each direction . Here , √ Vt : = V 1/2 t , mt/ √ Vt : = √ Vt −1 ·mt . By setting different mt , Vt and αt , we can derive different adaptive optimizers including MOMENTUM ( Polyak , 1964 ) , ADAGRAD ( Duchi et al. , 2011 ) , ADAM ( Kingma & Ba , 2015 ) , AMSGRAD ( Reddi et al. , 2018 ) and ADAHESSIAN ( Yao et al. , 2020 ) , etc . See Table 1 . 1.2 REGULARIZED OPTIMIZATION METHODS . Follow-the-regularized-leader ( FTRL ) ( McMahan & Streeter , 2010 ; McMahan et al. , 2013 ) has been widely used in click-through rates ( CTR ) prediction problems , which adds ` 1-regularization ( lasso ) to logistic regression and can effectively balance the performance of the model and the sparsity of features . The update formula ( McMahan et al. , 2013 ) is : xt+1 = arg min x g1 : t · x+ 1 2 t∑ s=1 σs‖x− xs‖22 + λ1‖x‖1 , ( 2 ) where g1 : t = ∑ t s=1gs , 1 2 ∑t s=1 σs‖x− xs‖22 is the strong convex term that stabilizes the algorithm and λ1‖x‖1 is the regularization term that produces sparsity . However , it doesn ’ t work well in DNN models since one input feature can correspond to multiple weights and lasso only can make single weight zero hence can ’ t effectively delete zeros features . To solve above problem , Ni et al . ( 2019 ) adds the ` 21-regularization ( group lasso ) to FTRL , which is named G-FTRL . Yang et al . ( 2010 ) conducts the research on a group lasso method for online learning that adds ` 21-regularization to the algorithm of Dual Averaging ( DA ) ( Nesterov , 2009 ) , which is named DA-GL . Even so , these two methods can not been applied to other optimizers . Different scenarios are suitable for different optimizers in the deep learning fields . For example , MOMENTUM ( Polyak , 1964 ) is typically used in computer vision ; ADAM ( Kingma & Ba , 2015 ) is used for training transformer models for natural language processing ; and ADAGRAD ( Duchi et al. , 2011 ) is used for recommendation systems . If we want to produce sparsity of the model in some scenario , we have to change optimizer which probably influence the performance of the model . 1.3 MOTIVATION . Eq . ( 1 ) can be rewritten into this form : xt+1 = arg min x mt · x+ 1 2αt ‖ √ Vt 1 2 ( x− xt ) ‖22 . ( 3 ) Furthermore , we can rewrite Eq . ( 3 ) into xt+1 = arg min x m1 : t · x+ t∑ s=1 1 2αs ‖Q 1 2 s ( x− xs ) ‖22 , ( 4 ) where m1 : t = ∑t s=1ms , ∑t s=1Qs/αs = √ Vt/αt . It is easy to prove that Eq . ( 3 ) and Eq . ( 4 ) are equivalent using the method of induction . The matrices Qs can be interpreted as generalized learning rates . To our best knowledge , Vt of Eq . ( 1 ) of all the adaptive optimization methods are diagonal for the computation simplicity . Therefore , we considerQs as diagonal matrices throughout this paper . We find that Eq . ( 4 ) is similar to Eq . ( 2 ) except for the regularization term . Therefore , we add the regularization term Ψ ( x ) to Eq . ( 4 ) , which is the sparse group lasso penalty also including ` 2- regularization that can diffuse weights of neural networks . The concrete formula is : Ψt ( x ) = G∑ g=1 ( λ1‖xg‖1 + λ21 √ dxg‖A 1 2 t x g‖2 ) + λ2‖x‖22 , ( 5 ) where λ1 , λ21 , λ2 are regularization parameters of ` 1 , ` 21 , ` 2 respectively , G is the total number of groups of weights , xg is the weights of group g and dxg is the size of group g. In DNN models , each group is defined as the set of outgoing weights from a unit which can be an input feature , or a hidden neuron , or a bias unit ( see , e.g. , Scardapane et al . ( 2016 ) ) . At can be arbitrary positive matrix satisfying At+1 At , e.g. , At = I . In Section 2.1 , we let At = ( ∑t s=1 Qgs 2αs + λ2I ) just for solving the closed-form solution directly , where Qgs is a diagonal matrix whose diagonal elements are part of Qs corresponding to xg . The ultimate update formula is : xt+1 = arg min x m1 : t · x+ t∑ s=1 1 2αs ‖Q 1 2 s ( x− xs ) ‖22 + Ψt ( x ) . ( 6 ) 1.4 OUTLINE OF CONTENTS . The rest of the paper is organized as follows . In Section 1.5 , we introduce the necessary notations and technical background . In Section 2 , we present the closed-form solution of Eq . ( 4 ) and the algorithm of general framework of adaptive optimization methods with sparse group lasso . We prove the algorithm is equivalent to adaptive optimization methods when regularization terms vanish . In the end , we give two concrete examples of the algorithm.1 In Section 3 , we derive the regret bounds of the method and convergence rates . In Section 4 , we validate the performance of new optimizers in the public datasets . In Section 5 , we summarize the conclusion . Appendices A-B list the details of GROUP ADAM and Group Adagrad respectively . Appendices C-F contain technical proofs of our main results and Appendix G includes the details of the empirical results of Section 4.4 . 1.5 NOTATIONS AND TECHNICAL BACKGROUND . We use lowercase letters to denote scalars and vectors , and uppercase letters to denote matrices . We denote a sequence of vectors by subscripts , that is , x1 , . . . , xt , and entries of each vector by an additional subscript , e.g. , xt , i . We use the notation g1 : t as a shorthand for ∑t s=1 gs . Similarly we write m1 : t for a sum of the first moment mt , and f1 : t to denote the function f1 : t ( x ) = ∑t s=1 fs ( x ) . Let Mt = [ m1 · · ·mt ] denote the matrix obtained by concatenating the vector sequence { mt } t≥1 andMt , i denote the i-th row of this matrix which amounts to the concatenation of the i-th component of each vector . The notation A 0 ( resp . A 0 ) for a matrix A means that A is symmetric and positive semidefinite ( resp . definite ) . Similarly , the notations A B and A B mean that A−B 0 andA−B 0 respectively , and both tacitly assume thatA andB are symmetric . Given A 0 , we write A 12 for the square root of A , the unique X 0 such that XX = A ( McMahan & Streeter ( 2010 ) , Section 1.4 ) . Let E be a finite-dimension real vector space , endowed with the Mahalanobis norm ‖ · ‖A which is denoted by ‖·‖A = √ 〈· , A·〉 as induced byA 0 . Let E∗ be the vector space of all linear functions on E . The dual space E∗ is endowed with the dual norm ‖ · ‖∗A = √ 〈· , A−1·〉 . Let Q be a closed convex set in E . A continuous function h ( x ) is called strongly convex on Q with norm ‖ · ‖H ifQ ⊆ dom h and there exists a constant σ > 0 such that for all x , y ∈ Q and α ∈ [ 0 , 1 ] we have h ( αx+ ( 1− α ) y ) ≤ αh ( x ) + ( 1− α ) h ( y ) − 1 2 σα ( 1− α ) ‖x− y‖2H . 1To fulfill research interest of optimization methods , we will release the code in the future . The constant σ is called the convexity parameter of h ( x ) , or the modulus of strong convexity . We also denote by ‖ · ‖h = ‖ · ‖H . Further , if h is differential , we have h ( y ) ≥ h ( x ) + 〈∇h ( x ) , y − x〉+ σ 2 ‖x− y‖2h . We use online convex optimization as our analysis framework . On each round t = 1 , . . . , T , a convex loss function ft : Q 7→ R is chosen , and we pick a point xt ∈ Q hence get loss ft ( xt ) . Our goal is minimizing the regret which is defined as the quantity RT = T∑ t=1 ft ( xt ) −min x∈Q T∑ t=1 ft ( x ) . ( 7 ) Online convex optimization can be seen as a generalization of stochastic convex optimization . Any regret minimizing algorithm can be converted to a stochastic optimization algorithm with convergence rate O ( RT /T ) using an online-to-batch conversion technique ( Littlestone , 1989 ) . In this paper , we assume Q ≡ E = Rn , hence we have E∗ = Rn . We write sTx or s · x for the standard inner product between s , x ∈ Rn . For the standard Euclidean norm , ‖x‖ = ‖x‖2 =√ 〈x , x〉 and ‖s‖∗ = ‖s‖2 . We also use ‖x‖1 = ∑n i=1 |x ( i ) | and ‖x‖∞ = max i|x ( i ) | to denote ` 1-norm and ` ∞-norm respectively , where x ( i ) is the i-th element of x .

Summary: This work studies the adaptive proximal gradient descent method, and specifically studies the group sparsity. To encourage the group sparsity, a regularizer which is a combination of $\ell_1$ norm, block $\ell_1$ norm and $\ell_2$ norm square is used. This paper gives the update rule of the proximal gradient with the specific regularizer. After proposing the update rule, the paper analyzes the convergence and regret guarantee of the algorithm.