On the Tunability of Optimizers in Deep Learning

There is no consensus yet on the question whether adaptive gradient methods like Adam are easier to use than non-adaptive optimization methods like SGD . In this work , we fill in the important , yet ambiguous concept of ‘ ease-of-use ’ by defining an optimizer ’ s tunability : How easy is it to find good hyperparameter configurations using automatic random hyperparameter search ? We propose a practical evaluation protocol for optimizer tunability that can form the basis for a fair optimizer benchmark . Evaluating a variety of optimizers on an extensive set of standard datasets and architectures , we find that Adam is the most tunable for the majority of problems , especially with a low budget for hyperparameter tuning . 1 INTRODUCTION . With the ubiquity of deep learning in various applications , a multitude of first-order stochastic optimizers ( Robbins & Monro , 1951 ) have been in vogue . They have varying algorithmic components like momentum ( Sutskever et al. , 2013 ) and adaptive learning rates ( Tieleman & Hinton , 2012 ; Duchi et al. , 2011 ; Kingma & Ba , 2015 ) . With all these choices , picking the optimizer is among the most important design decisions for machine learning practitioners . For this decision , the best possible generalization performance is certainly an important characteristic to be taken into account . However , we argue that in practice , an even more important characteristic is whether the best possible performance can be reached with the available resources . The performance of optimizers strongly depends on the choice of hyperparameter values such as the learning rate . In the machine learning research community , the sensitivity of models to hyperparameters has been of great debate recently , where in multiple cases , reported model advances did not stand the test of time because they can be explained by better hyperparameter tuning ( Lucic et al. , 2018 ; Melis et al. , 2018 ; Henderson et al. , 2018 ) . This has led to calls for using automatic hyperparameter optimization methods with a fixed budget for a fairer comparison of models ( Sculley et al. , 2018 ; Feurer & Hutter , 2019 ; Eggensperger et al. , 2019 ) . For industrial applications , automated machine learning ( AutoML , Hutter et al. , 2019 ) , which has automatic hyperparameter optimization as one of its key concepts , is becoming increasingly more important . In both cases , an optimization algorithm that achieves good performances with relatively little tuning effort is arguably substantially more useful than an optimization algorithm that achieves top performances , but reaches it only with a lot of careful tuning effort . Hence , we advocate that the performance obtained by an optimizer is not only the best performance obtained when using that optimizer , but also has to account for the cost of tuning its hyperparameters to obtain that performance , thus being dichotomous . We term this concept tunability in this paper . Despite the importance of this concept , there is no standard way of measuring tunability . Works that propose optimization techniques show their performance on various tasks as depicted in Table 1 . It is apparent that the experimental settings , as well as the network architectures tested , widely vary , hindering a fair comparison . The introduction of benchmarking suites like DEEPOBS ( Schneider et al. , 2019 ) have standardized the tested architectures , however , this does not fix the problem of selecting the hyperparameters themselves , and the effort expended in doing so . Previous studies treat tunability to be the best performance obtained on varying a hyperparameter ( Schneider et al. , 2019 ) or by measuring the improvement in performance by tuning a hyperparameter ( Probst et al. , 2019 ) , but do not take any cognizance to the intermediate performance during the tuning process . In this paper , we introduce a fair evaluation protocol for tunability based on automatic hyperparameter optimization , and simple evaluation measures that allow to compare the performance of optimizers under varying resource constraints . By evaluating on a wide range of 9 diverse tasks , we aim to contribute to the debate of adaptive vs. non-adaptive optimizers ( Wilson et al. , 2017 ; Shah et al. , 2018 ; Chen & Gu , 2018 ) . To reach a fair comparison , we experiment with several SGD variants that are often needed to reach good performance . Although a well-tuned SGD variant is able to reach the top performance in some cases , our overall results clearly favor adaptive gradient methods . We therefore conclude that there is substantial value in adaptive gradient methods . 2 MEASURING TUNABILITY . Given the dichotomy of the problem of tunability , we argue that it needs to take into account 1. how difficult it is to find a good hyperparameter configuration for the optimizer , 2. the absolute performance of the optimizer . To see why both are needed , consider Figure 1.a , which shows the performance in terms of loss of four different optimizers as a function of its only hyperparameter θ ( by assumption ) . If we only consider requirement # 1 , optimizer C would be considered the best , since every hyperparameter value is the optimum . However , its absolute performance is poor , making it of low practical value . Moreover , due to the same shape , optimizers A and B would be considered equally good , although optimizer A clearly outperforms B . On the other hand , if we only consider requirement # 2 , optimizers B and D would be considered equally good , although optimizer D ’ s optimum is harder to find . As we show in Section 5 , no existing definition of tunability takes both requirements into account . In the following , we present a formulation that does so . 2.1 PRELIMINARIES : HYPERPARAMETER OPTIMIZATION . We define hyperparameter optimization ( HPO ) ( Feurer & Hutter , 2019 ) as follows : Definition . LetM be an optimization algorithm with N hyperparameters ( θ1 , . . . , θN ) ∈ Θ . Let a specific instantiation ofM with θ ∈ Θ be denoted byMθ . Thus , given a datasetD = Dtrain ⋃ Dval , the following objective is minimized θ ? = arg min θ∈Θ L ( Mθ , Dval ) whereMθ is trained on Dtrain . In our work , we use L to be validation loss . We use L ( θ ) to refer to L ( Mθ , Dval ) for brevity . In our experiments , we use the time-tested Random Search ( Bergstra & Bengio , 2012 ) algorithm for HPO for simplicity . 2.2 DEFINING TUNABILITY . The ability to easily find good minima depends on the loss surface L itself . Thus , tunability is a characterization of the HPO ’ s loss function L. Here we present a quantification of this idea , which we illustrate with Figure 1.b . Let us assume that there are two optimizers E & F , both with hyperparameter θ , both of them used to minimize a function ( e.g . train a neural network ) . Let the loss functions of HPO be LE and LF respectively . As the figure shows , the minimum of LE is lower than that of LF ( denoted by θ ? E and θ ? F ) i.e . LE ( θ ? E ) < LF ( θ ? F ) . However , the minimum of LE is much sharper than that of LF , and Hyperparameter θ E xp ec te d lo ss Optimizer A Optimizer D Optimizer B Optimizer C 1.a : Illustration . It is important to consider both the absolute performance of optimizers as well as the tuning effort to get to good performances . Hyperparameter θ E xp ec te d lo ss L ( θ ) Optimizer F Optimizer E θ ? F θ ? E 1.b : Illustration . While optimizer E can achieve the best performance after careful tuning , optimizer F is likely to provide better performance under a constrained HPO budget . in most regions of the parameter space F performs much better than E. This makes it easier to find configurations that already perform well . This makes optimizer F an attractive option when we have no prior knowledge of the good parameter settings . The key-difference between the two interpretations is whether one prefers a ‘ good enough ’ performance through fewer hyperparameter configuration searches ( in the case of optimizer F ) , or whether one is willing to spend computational time to get the best possible performance ( θ ? E in the case of optimizer E ) . In this work , we search for the hyperparameter through an HPO like Random Search . Thus , the difference of the two interpretations of tunability lies whether one values results from late stages of the HPO process ( i.e . optimizer E is preferable due to better optimum ) more than results from early stages of the HPO process ( i.e . optimizer F is preferable ) . Motivated by these observations , we propose the following metric for tunability . ω-tunability ’ s Definition . Let ( θt , L ( θt ) ) be the incumbents ( best performance attained till t ) of the HPO algorithm at iteration t and T be the hyperparameter tuning budget . For wt > 0 ∀ t and∑ t wt < ∞ , we define ω-tunability as ω-tunability = T∑ t=1 ωtLt i.e , ω-tunability is a weighted sum of the incumbents L ( θt ) . In our experiments we use ∑ t ωt = 1 . By appropriately choosing the weights { ωt } , we can interpolate between our two notions of tunability . In the extreme case where we are only interested in the peak performance of the optimizer , we can set ωT = 1 and set the other weights to zero . In the opposite extreme case where we are interested in the `` one-shot tunability '' of the optimizer , we can set ω1 = 1 . In general , we can answer the question of `` How well does the optimizer perform with a budget of K iterations ? '' by setting ωi = 1i=K . While the above weighting scheme is intuitive , merely computing the performance after expending HPO budget of K does not consider the performance obtained after the previous K − 1 iterations i.e . we would like to differentiate the cases where a requisite performance is attained by tuning an optimizer for K iterations and another for K1 iterations , where K1 K. Therefore , we employ three additional weighting schemes . By setting ωi ∝ ( T − i ) , our first one puts more emphasis on the earlier stages of the hyperparameter tuning process . We term this weighting scheme Cumulative Performance-Early ( CPE ) . In contrast , the second weighting scheme , Cumulative Performance-Late ( CPL ) puts more emphasis on late stages of tuning , and thus on obtaining a better performance at a higher tuning cost : ωi ∝ i . As a intermediate of the two , we also report a uniform weighting Cumulative Performance-Uniform ( CPU ) : ωi = 1/T . 3 OPTIMIZERS AND THEIR HYPERPARAMETERS . 3.1 PARAMETERS OF THE OPTIMIZERS . To compare the tunability of adaptive gradient methods to non-adaptive methods , we choose the most commonly used optimizers from both the strata ; SGD and SGD with momentum for non-adaptive methods , and Adagrad and Adam for adaptive gradient methods . Since adaptive gradient methods are said to work well with their default hyperparameter values already , we additionally employ a default version of Adam where we only tune the initial learning rate and set the other hyperparameters to the values recommended in the original paper ( Kingma & Ba , 2015 ) ( termed AdamLR ) . Such a scheme has been used by Schneider et al . too . A similar argument can be made for SGD with momentum ( termed SGDM ) : thus we experiment with a fixed momentum value of 0.9 ( termed SGDMC ) . In addition to standard parameters in all optimizers , we consider weight decay with SGD too . SGD with weight decay can be considered as an optimizer with two steps where the first step is to scale current weights with the decay value , followed by a normal descent step ( Loshchilov & Hutter , 2019 ) . Thus we devise two additional experiments for SGD with weight-decay where we tune weight-decay along with momentum ( termed SGDMW ) , and one where we fix it to 10−5 ( termed SGDMCWC ) along with the momentum being fixed to 0.9 , which is the value for weight decay we found to be consistently better through HPO . The full list of optimizers we consider is provided in Table 4 Manually defining a specific number of epochs can be biased towards one optimizer , as one optimizer may reach good performance in the early epochs of a single HPO iteration , whereas another may reach higher peaks more slowly . In order to alleviate this , it would be possible to add the number of training epochs as an additional hyperparameter to be searched . Since this would incur even higher computational cost , we instead use a validation set performance as stopping criterion . Thus we stop training when the validation loss plateaus for more than 2 epochs or if the number of epochs exceeds the predetermined maximum number as set in DEEPOBS .

Summary: This paper introduces a simple measure of tunability that allows to compare optimizers under varying resource constraints. The tunability of the optimizer is a weighted sum of best performance at a given budget. The authors found that in a setting with low budget for hyperparameter tuning, tuning only Adam optimizer’s learning rate is likely to be a very good choice; it doesn’t guarantee the best possible performance, but it is evidently the easiest to find well-performing hyperparameter configurations.