Adaptive Hierarchical Hyper-gradient Descent

1 INTRODUCTION . The basic optimization algorithm for training deep neural networks is the gradient descent method ( GD ) , which includes stochastic gradient descent ( SGD ) , mini-batch gradient descent , and batch gradient descent . The model parameters are updated according to the first-order gradients of the empirical risks with respect to the parameters being optimized , while back-propagation is implemented for calculating the gradients of parameters ( Ruder , 2016 ) . Naïve gradient descent methods apply fixed learning rates without any adaptation mechanisms . However , considering the change of available information during the learning process , SGD with fixed learning rates can result in inefficiency and requires a large amount of computing resources in hyper-parameter searching . One solution is to introduce a learning rate adaptation . This idea can be traced back to the work on gain adaptation for connectionist learning methods ( Sutton , 1992 ) and related extensions for non-linear cases ( Schraudolph , 1999 ; Yu et al. , 2006 ) . In recent years , optimizers with adaptive updating rules were developed in the context of deep learning , while the learning rates are still fixed in training . The proposed methods include AdaGrad ( Duchi et al. , 2011 ) , RMSProp ( Tieleman and Hinton , 2012 ) , and Adam ( Kingma and Ba , 2015 ) . In addition , there are optimizers aiming to address the convergence issue in Adam ( Reddi et al. , 2018 ; Luo et al. , 2018 ) and to rectify the variance of the adaptive learning rate ( Liu et al. , 2019 ) . Other techniques , such as Lookahead , can also achieve variance reduction and stability improvement with negligible extra computational cost ( Zhang et al. , 2019 ) . Even though the adaptive optimizers with fixed learning rates can converge faster than SGD in a wide range of tasks , the updating rules are designed manually while more hyper-parameters are introduced . Another idea is to use objective function information and update the learning rates as trainable parameters . These methods were introduced as automatic differentiation , where the hyper-parameters can be optimized with backpropagation ( Maclaurin et al. , 2015 ; Baydin et al. , 2018 ) . As gradient-based hyper-parameter optimization methods , they can be implemented as an online approach ( Franceschi et al. , 2017 ) . With the idea of auto-differentiation , learning rates can be updated in real-time with the corresponding derivatives of the empirical risk ( Almeida et al. , 1998 ) , which can be generated to all types of optimizers for deep neural networks ( Baydin et al. , 2017 ) . Another step size adaptation approach called “ L4 ” , is based on the linearized expansion of the loss functions , which rescales the gradient to make fixed predicted progress on the loss ( Rolinek and Martius , 2018 ) . Furthermore , by addressing the issue of poor generalization performance of adaptive methods , dynamically bound for gradient methods was introduced to build a gradual transition between adaptive approach and SGD ( Luo et al. , 2018 ) . Another set of approaches train an RNN ( recurrent neural network ) agent to generate the optimal learning rates in the next step given the historical training information , known as “ learning to learn ” ( Andrychowicz et al. , 2016 ) . This approach empirically outperforms hand-designed optimizers in a variety of learning tasks , but another study has shown that it may not be effective for long horizons ( Lv et al. , 2017 ) . The generalization ability of this approach can be improved by using meta training samples and hierarchical LSTMs ( long short-term memory ) ( Wichrowska et al. , 2017 ) . Beyond the adaptive learning rate , learning rate schedules can also improve the convergence of optimizers , including time-based decay , step decay , exponential decay ( Li and Arora , 2019 ) . The most fundamental and widely applied one is a piece-wise step-decay learning rate schedule , which could vastly improve the convergence of SGD and even adaptive optimizers ( Luo et al. , 2018 ; Liu et al. , 2019 ) . It can be further improved by introducing a statistical test to determine when to apply step-decay ( Lang et al. , 2019 ; Zhang et al. , 2020 ) . Also , there are works on warm-restart ( O ’ donoghue and Candes , 2015 ; Loshchilov and Hutter , 2017 ) , which could improve the performance of SGD anytime when training deep neural networks . We find that the existing gradient or model-based learning rate adaptation methods including hypergradient descent , L4 and learning to learn only focus on global adaptation , which could be further extended to multi-level cases . That focus aims to introduce locally shared adaptive learning rates such as the layer-wise learning rate and parameter-wise learning rate and considers all levels ’ information in determining the updating step-size for each parameter . The main contribution of our study can be summarized as follows : • We introduce hierarchical learning rate structures for neural networks and apply hypergradient descent to obtain adaptive learning rates at different levels . • We introduce a set of regularization techniques for learning rates to address the balance of global and local adaptations and show the relationship with weighted combinations . • We propose an algorithm implementing the combination of adaptive learning rates at multiple levels for model parameter updating . 2 MULTI-LEVEL ADAPTATION METHODS . 2.1 LAYER-WISE , UNIT-WISE AND PARAMETER-WISE ADAPTATION . In the paper on hyper-descent ( Baydin et al. , 2017 ) , the learning rate is set to be a scalar . However , to make the most of learning rate adaptation , in this study , we introduce layer-wise or even parameterwise updating rules , where the learning rate αt in each iteration time step is considered to be a vector ( layer-wise ) or even a list of matrices ( parameter-wise ) . For the sake of simplicity , we collect all the learning rates in a vector : αt = ( α1 , t , ... , αN , t ) T . Correspondingly , the objective f ( θ ) is a function of θ = ( θ1 , θ2 , ... , θN ) T , collecting all the model parameters . In this case , the derivative of the objective function f with respect to each learning rate can be written as ∂f ( θt−1 ) ∂αi , t−1 = ∂f ( θ1 , t−1 , ... , θN , t−1 ) ∂αi , t−1 = N∑ j=1 ∂f ( θ1 , t−1 , ... , θN , t−1 ) ∂θj , t−1 ∂θj , t−1 ∂αi , t−1 , ( 1 ) where N is the total number of all the model parameters . Eq . ( 1 ) can be generalized to groupwise updating , where we associate a learning rate with a special group of parameters , and each parameter group is updated according to its only learning rate . Notice that although there is a dependency between αt−1 and θt−2 with : αt−1 = αt−2 − β∇f ( θt−2 ) , where β is the updating rate of hyper-gradient descent , we consider that αt−1 is calculated after θt−2 and thus a change of αt−1 will not result in a change of θt−2 . Assume θt = u ( Θt−1 , α ) is the updating rule , where Θt = { θs } ts=0 and α is the learning rate , then the basic gradient descent method for each group i gives θi , t = u ( Θt−1 , αi , t−1 ) = θi , t−1 − αi , t−1∇θif ( θt−1 ) . Hence for gradient descent ∂f ( θt−1 ) ∂αi , t−1 = ∇θif ( θt−1 ) T∇αi , t−1u ( Θt−1 , αi , t−1 ) = −∇θif ( θt−1 ) T∇θif ( θt−2 ) . ( 2 ) Here αi , t−1 is a scalar with index i at time step t − 1 , corresponding to the learning rate of the ith group , while the shape of ∇θif ( θ ) is the same as the shape of θi . We particularly consider two special cases : ( 1 ) In layer-wise adaptation , θi is the weight matrix of ith layer , and αi is the particular learning rate for this layer . ( 2 ) In parameter-wise adaptation , θi corresponds to a certain parameter involved in the model , which can be an element of the weight matrix in a certain layer . 2.2 REGULARIZATION ON ADAPTIVE LEARNING RATES . The selection of adaptation level should depend on a case-by-case basis . Global or parameter-wise adaptation is usually not the optimal choice across all circumstances . Recall that for deep neural networks , we typically use a relatively large architecture with regularization . This idea can also be applied to learning rate space with parameter structure . To address over-parameterization in implementing lower-level learning rate adaptation , we introduce regularization on learning rates to control the flexibility . First , for layer-wise adaptation , we can add the following regularization term to the loss function Llr_reg_layer = λlayer ∑ l ( αl − αg ) 2 , ( 3 ) where l is the indices for each layer , λlayer is the layer-wise regularization coefficient , αl and αg are the layer-wise and global-wise adaptive learning rates . A larger λlayer can push each layer ’ s learning rate towards the global learning rate across all the layers . Given a particular αg , t , the gradient of the loss function with respect to the learning rate αl in layer l can be written as ∂Lfull ( θ , α ) ∂αl , t = ∂Lmodel ( θ , α ) ∂αl , t + ∂Llr_reg ( θ , α ) ∂αl , t = ∇θlf ( θt−1 ) T∇αl , t−1u ( Θt−2 , αt−1 ) + 2λlayer ( αl , t − αg , t ) . ( 4 ) Notice that the time step index of layer-wise regularization term is t rather than t− 1 , which ensures that we push the layer-wise learning rates towards the corresponding global learning rates of the current step t. Denoting by hl , t−1 = −∇θlf ( θt−1 ) T∇θlu ( Θt−2 , αl , t−1 ) , then the updating rule for learning rates can be written as αl , t = αl , t−1 − β ∂Lfull ( θ , α ) ∂αl , t = αl , t−1 − β ( −hl , t−1 + 2λlayer ( αl , t − αg , t ) ) . ( 5 ) Eq . ( 5 ) has a close form solution but only applicable in the two-levels case . However , there is an extra hyper-parameter λlayer to be tuned . In addition , when there are more levels , components of learning rates at different levels can be interdependent . To construct a workable updating scheme for Eq . ( 5 ) , we replace αl , t and αg , t with their relevant approximations . We take the strategy of using their updated version without considering regularization , i.e. , α̂l , t = αl , t−1 + βhl , t−1 and α̂g , t = αg , t−1 + βhg , t−1 , where hg , t−1 = −∇θf ( θt−1 ) T∇αg , t−1u ( Θt−2 , αg , t−1 ) is the global h for all parameters . Here we regard α̂l , t and α̂g , t as the “ virtual ” layer-wise and global-wise learning rates for time step t and taking them into the right-hand side of Eq . ( 5 ) gives the new updating rule as follows α∗l , t = αl , t−1 + βhl , t−1 − 2βλlayer ( α̂l , t − α̂g , t ) = ( 1− 2βλlayer ) α̂l , t + 2βλlayerα̂g , t . ( 6 ) Notice that in Eq . ( 6 ) , the two terms are actually a weighted average of the layer-wise learning rate α̂l , t and global learning rate α̂g , t at the current time step . Since we hope to push the layer-wise learning rates towards the global one , the parameters should meet the constraint : 0 < 2βλlayer < 1 , and thus they can be optimized using hyper-parameter searching within a bounded interval as well as gradient-based hyper-parameter optimizations . We can also consider the case where three levels of learning rate adaptations are involved , including global-wise , layer-wise , and parameter-wise adaptation . If we introduce two more regularization terms to control the variation of parameter-wise learning rate with respect to layer-wise learning rate and global learning rates , the regularization loss can be written as Llr_reg_para = λlayer ∑ l ( αl − αg ) 2 + λpara_layer ∑ l ∑ p ( αpl − αl ) 2 + λpara ∑ l ∑ p ( αpl − αg ) 2 , where αpl is the learning rate for the p-th parameter inside layer l. The second and third terms push each parameter-wise learning rate towards the layer-wise learning rate and the global learning rates , respectively . Like the two-level case , the updating rule with this three-level regularization can be approximated by the weighted combination of three components under “ virtual approximation ” . The detail of the updating rule for the three-levels case is given by Algorithm 1 in Section 2.3 . We also provided a discussion on the bias of implementing “ virtual approximation ” in Appendix A.1 . In general , we can organize all the learning rates in a tree structure . For example , in the three-level case above , αg will be the root node , while { αl } are the children node at level 1 of the tree and { αlp } are the children node of αl as leaf nodes at level three of the tree . In a general case , we assume there are L levels in the tree . Denote the set of all the paths from the root node to each of leave nodes as P and a path is denoted by p = { α1 , α2 , ... , αL } where α1 is the root node , and αL is the left node on the path . On this path , denote ancestors ( i ) all the ancestor nodes of αi along the path , i.e. , ancestors ( i ) = { α1 , ... , αi−1 } . We will construct a regularizer to push αi towards each of its parents . Then the regularization can be written as Llr_reg = ∑ p∈P ∑ αi∈p ∑ αj∈ancestor ( i ) λij ( αi − αj ) 2 . ( 7 ) Under this pair-wise L2 regularization , the updating rule for any leave node learning rate αL can be given by the following theorem whose proof is provided in Appendix A.2 . Theorem 1 . Under virtual approximation , the effect of applying pair-wise L2 regularization Eq . ( 7 ) results in performing a weighted linear combination of virtual learning rates at different levels α∗L = ∑L j=1 γjα̂j with ∑L j=1 γj = 1 , where each component α̂j is calculated by assuming no regularization . Remarks : Theorem 1 actually suggests that a similar updating rule can be obtained for the learning rate at any level on the path . All these have been demonstrated in Algorithm 1 for the three-level case .

Summary: Setting appropriate learning rate for network optimization is an important task in deep learning applications. This paper investigates the setting of learning rates for network parameters in different levels, e.g., individual parameter, each layer and global levels. By setting the constraints on the learning rates at multiple scales, the paper derived a hierarchical learning rate setting approach, which is the combination of adaptive learning rates at different levels.