Combining adaptive algorithms and hypergradient method: a performance and robustness study

1 INTRODUCTION . Many new algorithms have been proposed in recent years for the minimization of unconstrained nonconvex functions such as the loss used in deep neural networks . A critical parameter of all these methods is the stepsize . A poor choice for that stepsize can either lead to very slow training or divergence . Worse , in many cases , the stepsize leading to the fastest minimization is the largest one achieving convergence , making the search difficult . The dramatic effect of that tuning motivated the development of a range of optimization methods trying to integrate temporal metrics to adapt the stepsize for each parameter during optimization ( Duchi et al. , 2011 ; Zeiler , 2012 ; Tieleman & Hinton , 2012 ; Kingma & Ba , 2014 ) . In particular , ADAM ( Kingma & Ba , 2014 ) has become the default choice for many researchers and practitioners . One reason for this success is that these methods use a stepsize that is approximately normalized , that is the optimal stepsize potentially varies less across datasets and architectures than the optimal stepsize for non-adaptive methods . However , Wilson et al . ( 2017 ) analyzed in depth the impact of adaptive methods on both training and generalization and showed that stochastic gradient methods with a carefully tuned stepsize could reach a lower generalization error than methods optimizing one stepsize per parameter . In order to achieve this result , a well tuned learning rate along with a suitable decaying schedule was required . A recent approach for tuning the learning rate online was proposed by Baydin et al . ( 2018 ) . This method , called hypergradient descent ( HD ) , does not require setting a decay schedule ahead of time . One may thus wonder if , by automatically tuning the stepsize , such a technique would remove the last remaining advantage of adaptive methods , i.e . easier tuning . Our work relates this technique to the recent criticism made about the adaptive gradient methods ( Wilson et al. , 2017 ) and reconsider the value of these methods compared to their non-adaptive counterparts . More precisely , this paper aims at extending the analysis of Wilson et al . ( 2017 ) in the following ways : • How competitive is the recent online hypergradient scheme proposed by Baydin et al . ( 2018 ) compared to the offline scheme of Wilson et al . ( 2017 ) ? • Does this online scheme change the conclusions of Wilson et al . ( 2017 ) ? • Does this online scheme remove the need for fine-tuning the optimizer ’ s hyperparameters , thereby removing the advantage of ADAM over stochastic gradient with momentum ? • What is the sensitivity of the learning rate schedule to a suboptimal choice of the hyperpa- rameters ? The last point is often overlooked in the study of optimization methods . While investigating which training conditions bring the best performance led to significant progress in the field , the effort needed to have an optimizer perform at its best should be taken into account when evaluating the performance . Consider the following question : given a desired level of performance and limited computational ressources , which optimization method should be prefered and how should it be tuned ? By this work , we would like to emphasize the value of tuning for gradient based methods , and what it can reveal about them . 2 BACKGROUND . We now review several techniques for stepsize selection . These techniques may either try to find a single global stepsize or one stepsize for each parameter in the model . 2.1 ADAPTIVE STEPSIZE . A first class of methods are the adaptive methods . To compute the parameter updates , these methods multiply the gradient with a matrix . When this matrix is diagonal , this is equivalent to using a distinctive stepsize per parameter . One of the first such method is Newton ’ s method , multiplying the gradient with the inverse of the Hessian to minimize the local quadratic approximation to the loss . Similarly , Amari ( 1998 ) leveraged the Fisher information matrix to precondition the gradient . Later , Le Roux et al . ( 2008 ) used the covariance of the gradients to move less in directions of high uncertainty . A diagonal version of that technique , which adapts the learning rate on a per parameter basis , ADAGRAD , was proposed by Duchi et al . ( 2011 ) . Subsequently , many works sought to improve upon ADAGRAD , such as RMSprop ( Tieleman & Hinton , 2012 ) , ADADELTA ( Zeiler , 2012 ) and ADAM ( Kingma & Ba , 2014 ) . Other methods ( Schaul et al. , 2013 ) adapt the stepsize to minimize an estimate of the expected loss . 2.2 LEARNING RATE SCHEDULE . Despite learning a stepsize per parameter , adaptive methods still require a global stepsize , which might change during the course of optimization . We distinguish two classes of stepsize schedules : offline schedules , which are set ahead of time , and online schedules , which depend on the optimization trajectory . OFFLINE LEARNING RATE SCHEDULES Offline schedules define before the optimization , for each iteration t , the value of the stepsize αt . The seminal work of Robbins & Monro ( 1951 ) advocated for a schedule αt = O ( 1/t ) , a rate often too slow in practice . Another decay schedule proposed by Bottou et al . ( 2018 ) is to halve the stepsize every time t = 2k . The overall rate remainsO ( 1/t ) but this method seems to work better in practice . A more aggressive schedule was proposed by Wilson et al . ( 2017 ) where the stepsize is halved every fixed number of iterations . While this leads to an exponential decay of the stepsize , precluding the method from reaching a local optimum from any initialization , this led them to the best results . In our work , we will call lr-decay the schedule where the stepsize is divided by 2 every 25 epochs . ONLINE LEARNING RATE SCHEDULES In contrast to offline schedules , online schedules tune the stepsize based on the dynamics of the optimization . Though extremely appealing , these methods have had little success , mostly because of their brittleness . Recently , however , Baydin et al . ( 2018 ) rediscovered the hypergradient update rule for the stepsize , which was originally suggested by Almeida et al . ( 1998 ) . The optimisation of the stepsize can be framed as an iterative process that increases the stepsize when the last two gradients are aligned , suggesting that an acceleration is possible , and decreases it otherwise . It requires no extra gradient computation and only needs a copy of the current gradient to be held in memory for the following iteration . More precisely , assume we want to minimize a parametric function f : Rd → R with d the number of parameters . Let us denote θt the parameter iterate at timestep t. A gradient descent update rule can be written : θt = θt−1 − α∇θf ( θt−1 ) = u ( Θt−1 , α ) where Θt = { θi } ti=0 and α is the learning rate . The goal is to move αt−1 toward the value that minimizes the expected objective function value at the next iteration , E [ f ( u ( Θt , α ) ) ] . Taking the derivative with respect to α , we end up , under suitable assumptions , with the following learning rate update rule αt = αt−1 − β∇̃θf ( θt−1 ) > ∇αu ( Θt−2 , αt−1 ) where ∇̃θf ( θ ) is the noisy gradient of f . The authors observed that HD decreases the sensitivity to α0 , the initial learning rate value , and that tuning only β is sufficient . In the following section , we will see whether this lower sensitivity comes with the ability to recover nearly optimal optimization trajectory , or if it would still benefit from some tuning . 3 THE VALUE OF ONLINE LEARNING RATE ADAPTATION . Now , we focus on comparing hypergradient to the fixed decay scheme of Wilson et al . ( 2017 ) . We show that , i ) hypergradient can significantly benefit from a fine tuning of its hyperparameters , and by doing so , ii ) it replicates , in most cases , the training performance of the best offline learing rate adaptation scheme and , that iii ) stochastic gradient with hypergradient generalizes better than ADAM , extending Wilson et al . ( 2017 ) conclusions . 3.1 EXPERIMENTAL SETUP . We consider three gradient methods : SGD , SGD with Nesterov momentum ( Sutskever et al. , 2013 ) ( SGDN ) and ADAM . For each of these methods , we tried both hypergradient ( SGD-HD , SGDN-HD , ADAM-HD ) and the offline fixed decay ( SGD-decay , SGDN-decay , ADAM-decay ) . We study the performance of these six algorithms on three tasks : MNIST classification task using a feedforward neural network with two hidden layers of 1000 hidden units each , CIFAR-10 classification task using the VGG16 architecture from the Torch blog ( Zagoruyko , 2015 ) , and CIFAR-10 classification task using ResNet18 ( He et al. , 2016 ) . A grid search was performed to optimize the learning rate for each of these methods . The same grid of values was used for SGD and SGDN . For ADAM , another grid of comparable size was used . The scale of each grid was spread around the scale that is commonly known to be suitable for each of these optimizers . For the hypergradient method ( HD ) , the grid search was conducted over both the initial learning rate α0 and the hypergradient learning rate β . However , for the learning rate fixeddecay ( lr-decay ) , we kept the values of the decay factor and the decay frequency used in Wilson et al . ( 2017 ) , which they have already optimized . The list of hyperparameters tried may be found in the appendix . For momentum methods , we used µ = 0.9 . For Adam , we used the default setting with β1 = 0.9 , β2 = 0.999 and = 10−8 . L2 regularization with a coefficient of 10−4 was used for all our experiments . Each experiement was conducted five times , using the same random initialization for all optimizers each time . CIFAR-10 was trained for 250 epochs and MNIST for 200 epochs . We now present our results . 3.2 Hypergradient AND THE NEED OF TUNING Before comparing hypergradient and lr-decay , we give a sense of how much hypergradient can boost the optimizers we are considering . We will later compare the best performing configurations to lr-decay . Baydin et al . ( 2018 ) claim that hypergradient “ significantly reduces the need of the manual tuning of the initial learning rate for these commonly used algorithms ” . For an arbitraty initial learning rate , hypergradient “ consistently brings the loss trajectory closer to the optimal one ” . As can be seen in Figure 1 , a slightly better tuning improves drastically the performance . Thus , even if hypergradient decreases the sensitivity to the inital learning rate for a fixed value of β , tuning both parameters remains essential . Figure 1 gives the learning curves of hypergradient optimizers with tuned α0 and β . Compared to the results of Baydin et al . ( 2018 ) , we now see that SGD-HD and SGDN-HD are competitive and able to outperform ADAM-HD . Our tuned configuration for ADAM-HD actually coincindes with the one given in Baydin et al . ( 2018 ) .

Summary: The paper reports the results of testing several stepsize adjustment related methods including  vanilla SGD, SGD with Neserov momentum, and ADAM. Also, it compares those methods with hypergradient and without. The paper reports several interesting results. For instance, they found hypergradient method on common optimizers doesn't perform better that the fixed exponential decay method propose by Wilson et al. (2017).