Hyper-Regularization: An Adaptive Choice for the Learning Rate in Gradient Descent

1 INTRODUCTION . The automatic choice of the learning rate remains crucial in improving the efficiency of gradient descent algorithms , especially for solving nonconvex optimization problems . It is desirable to adaptively update the learning rate during the training process with a certain strategy . The convergence guarantees of such a strategy in theories usually require the Lipschitz constant or smoothness constant of the objective function to be explicitly known ( Nesterov , 2013 ; Bubeck et al. , 2015 ) , which is inaccessible in most cases , e.g. , in deep neural networks . Using the received gradient information to adjust the current learning rate is a natural approach . In particular , the Steepest Descent uses the received gradient direction and an exact or inexact line search to obtain proper learning rates . Another important idea is to approximate second-order methods like Newton method ( Nocedal & Wright , 2006 ) and Quasi-Newton Methods ( Liu & Nocedal , 1989 ) . Along this idea , for example , the Barzilai-Borweinin ( BB ) method ( Barzilai & Borwein , 1988 ) in classical optimization and AdaGrad ( Duchi et al. , 2011 ) in online learning have been proposed . In this paper we propose a novel framework to learn the learning rate that we call HyperRegularization . More specifically , we regard the learning rate as a hyperparameter and cast its adaptive choice with the parameter training into a joint process . We formulate this process as a maxmin framework by imposing a regularizer on the hyperparameter . Furthermore , we demonstrate that AdaGrad and WNGrad ( Wu et al. , 2018 ) can be derived using a streamlined scheme based on Hyper-Regularization . In addition to solving the saddle point problem exactly , we also provide an alternating strategy to solve the problem approximately . We respectively give theoretical analysis for these two updating rules in full batch setting and online learning setting . Specifically , our results of runtime bounds in full batch setting and regret bounds in online learning setting are comparable to the best known bound in corresponding settings and indicate our algorithms converge for any initial learning rate . 1.1 RELATED WORK . Steepest Descent uses the received gradient direction and an exact or inexact line search to obtain proper learning rates . Although Steepest Descent uses the direction that descends most and the best learning rate that gives the most function reduction , Steepest Descent may converge very slow for convex quadratic functions when the Hessian matrix is ill-conditioned ( see , Yuan , 2008 ) . In practice , some line search conditions such as Goldstein conditions or Wolfe conditions ( see , Fletcher , 2013 ) can be applied to compute the learning rate . In online or stochastic settings , one observes stochastic gradients rather than exact gradients and line search methods become less effective . The Barzilai-Borwein method ( Barzilai & Borwein , 1988 ) which was motivated by quasi-Newton methods presents a surprising result that it could lead to superlinear convergence in convex quadratic problem of two variables . Although numerical results often show the Barzilai-Borwein method converges superlinearly in solving nonlinear optimization problem , no superlinear convergence results have been established even for an n-dimensional strictly convex quadratic problem with the order n > 2 ( Barzilai & Borwein , 1988 ; Dai , 2013 ) . In minimizing the sum of cost functions and stochastic setting , SGD-BB proposed by Tan et al . ( 2016 ) takes the average of the stochastic gradients in one epoch as an estimation of the full gradient . But this approach can not directly be applied to online learning settings . In online convex optimization ( Zinkevich , 2003 ; Shalev-Shwartz et al. , 2012 ; Hazan et al. , 2016 ) , AdaGrad ( Duchi et al. , 2011 ) adapts the learning rate on per parameter basis dynamically . Intuitively , AdaGrad constructs approximation to the Hessian with diagonal of accumulated outer products of gradients . This leads to many variants such as RMSProp ( Tieleman & Hinton , 2012 ) , AdaDelta ( Zeiler , 2012 ) , Adam ( Kingma & Ba , 2015 ) , etc . Additionally , Cruz ( 2011 ) analyzed Adaptive Stochastic Gradient Descent ( ASGD ) which is a generalization of Kesten ’ s accelerated stochastic approximation algorithm ( Kesten et al. , 1958 ) for the high-dimensional case . ASGD uses a monotone decreasing function with respect to a time variable to get learning rates . Recently , Baydin et al . ( 2018 ) proposed Hyper-Gradient Descent to learn the global learning rate in SGD , SGD with Nesterov momentum and Adam . Hyper-Gradient Descent can be viewed as an approximate line search method in the online learning setting and it uses the update rule for the previous step to optimize the leaning rate in the current step . However , HyperGradient Descent has no theoretical guarantee . It is worth mentioning that Gupta et al . ( 2017 ) proposed a framework named Unified Adaptive Regularization from which AdaGrad and Online Newton Step ( Hazan et al. , 2007 ) can be derived . However , Unified Adaptive Regularization gives an approach for approximating the Hessian matrix in second order methods . 1.2 NOTATION . Before introducing our approach , we present the notation . We denote the set { x > 0 : x ∈ R } by R++ . For two vectors a , b ∈ Rd , we use a/b to denote element-wise division , a ◦ b for elementwise product ( the symbol ◦ will be omitted in the explicit context ) , an = ( an1 , an2 , . . . , and ) , and a ≥ b if aj ≥ bj for all j . Let 1 be the vector of ones with an appropriate size , and diag ( β ) be a diagonal matrix with the elements of the vector β on the main diagonal . In addition , we define ‖a‖A = √ 〈a , Aa〉 . Given a set X ⊆ Rd , a function f : X → R is said to satisfy f ∈ C1,1L ( X ) if f is continuously differentiable on X , and the derivative of f is Lipschitz continuous on X with constant L : ‖∇f ( x ) −∇f ( y ) ‖2 ≤ L‖x− y‖2 . More general definition can be found in Nesterov ( 2013 ) . 1.3 PROBLEM STATEMENT . For an online learning problem , a learner faces a sequence of convex functions { ft } with the same domain X ⊆ Rd , receives ( sub ) gradient information gt ∈ ∂ft ( xt ) at each step t , and predicts a point xt+1 ∈ X . Our theoretical analysis is based on two settings : full batch setting and online learning setting . A full batch setting ( or optimization setting ) is to optimize a certain function F with exact gradient at each step , i.e. , ft = F . In this setting we assume F ∈ C1,1L but it is not necessarily convex . We analyze the convergence of our algorithms by capping the runtime T such that the minimum value of the norm of received gradients so far is less than a given error accuracy ε , that is , min t=0 : T−1 ‖∇F ( xt ) ‖22 ≤ ε . In online learning settings , our analysis follows from Duchi et al . ( 2011 ) and Kingma & Ba ( 2015 ) . We only assume that ft ’ s are convex and try to give an upper bound for the regret R ( T ) = T−1∑ t=0 ft ( xt ) − min x∈X T−1∑ t=0 ft ( x ) . ( 1 ) 2 HYPER-REGULARIZATION . Following the setting in AdaGrad ( Duchi et al. , 2011 ) , we consider a generalization of the standard ( sub ) gradient descent as xt+1 = Π diag ( βt ) 1/2 X ( xt − diag ( βt ) −1/2gt ) = arg min x∈X ∥∥∥xt − diag ( βt ) −1/2gt∥∥∥2 diag ( βt ) 1/2 . ( 2 ) This procedure can be viewed as the minimization problem : min x∈X 〈gt , x− xt〉+ 1 2 ‖x− xt‖2diag ( βt ) . ( 3 ) To derive our hyper-regularization approach , we then formulate this minimization problem as a saddle point problem by adding a hyper-regularizer about the difference between the new learning rate β and an auxiliary vector ηt . Accordingly , we have max β∈Bt min x∈X Ψt ( x , β ) , 〈gt , x− xt〉+ 1 2 ( ‖x− xt‖2diag ( β ) −D ( β , ηt ) ) , ( 4 ) where D ( β , η ) is defined as our hyper-regularizer and Bt is a subset in Rd . We solve the saddle point problem for new predictor and new learning rate . Our framework stems from the work of Daubechies et al . ( 2010 ) , where the authors adjust the weights of the weighted least squares problem by solving an extra objective function which added a regularizer about the weights to origin objective function . The following subsections will explain some details about our framework . 2.1 THE ϕ-DIVERGENCE It is reasonable to choose a distance function to measure the difference between β and η . In this paper , we use the ϕ-divergence1 as our hyper-regularizer . Definition 1 ( ϕ-divergence ) . Let ϕ : R++ → R be a differentiable strongly convex function in R++ such that ϕ ( 1 ) = ϕ′ ( 1 ) = 0 , where ϕ′ is the derivative function of ϕ . For such a function ϕ , the function Dϕ : Rd++ × Rd++ → R , which is define by Dϕ ( u , v ) , d∑ j=1 vjϕ ( uj/vj ) , is referred to as the ϕ-divergence . Remark . Note that convex function ϕ with ϕ ( 1 ) = ϕ′ ( 1 ) = 0 satisfies ϕ ( z ) ≥ 0 for all z > 0 , thus Dϕ ( u , v ) ≥ 0 for all u , v ∈ Rd++ , with equality iff u = v. Remark . For any convex function f , ϕ ( z ) = f ( z ) − f ′ ( 1 ) ( z − 1 ) − f ( 1 ) is a proper function for our ϕ-divergence . Remark . In our framework , in order to solve the problem ( 4 ) feasibly , we always assume that limz→+∞ ϕ ′ ( z ) = +∞ . 1Comparing with Bregman divergence , ϕ-divergence requires nonnegative β and ηt which is apparent for learning rates . If one only requires ϕ to be convex and ϕ ( 1 ) = 0 , the resulting distance function Dϕ is called a f -divergence ( Liese & Vajda , 1987 ; 2006 ) . The f -divergence has been widely applied in statistical machine learning ( e.g. , Nguyen et al. , 2009 ) . Using the ϕ-divergence as our hyper-regularizer , we can rewrite the problem ( 4 ) as max β∈Bt min x∈X Ψt ( x , β ) , g > t ( x− xt ) + 1 2 ‖x− xt‖2diag ( β ) − 1 2 Dϕ ( β , ηt ) , or ( 5 ) max β∈Bt min x∈X Ψt ( x , β ) , d∑ j=1 gt , j ( xj − xt , j ) + 1 2 ( βj ( xj − xt , j ) 2 − ηt , jϕ ( βj/ηt , j ) ) . ( 6 ) The form of problem ( 6 ) implies that we can solve the problem for each dimension separately , and only a few extra calculations are required for each step .

Summary: This paper presents a method for adaptively tuning the learning rate in gradient descent methods. The authors consider the formulation of each gradient descent update as a quadratic minimization problem and they propose adding a phi-divergence between the learning rate that would be used and an auxiliary vector. The authors also propose adding a maximization over all learning rates in the update.