On Empirical Comparisons of Optimizers for Deep Learning

1 INTRODUCTION . The optimization algorithm chosen by a deep learning practitioner determines the training speed and the final predictive performance of their model . To date , there is no theory that adequately explains how to make this choice . Instead , our community relies on empirical studies ( Wilson et al. , 2017 ) and benchmarking ( Schneider et al. , 2019 ) . Indeed , it is the de facto standard that papers introducing new optimizers report extensive comparisons across a large number of workloads . Therefore , to maximize scientific progress , we must have confidence in our ability to make empirical comparisons between optimization algorithms . Although there is no theory guiding us when comparing optimizers , the popular first-order optimizers form a natural inclusion hierarchy . For example , ADAM ( Kingma and Ba , 2015 ) and RMSPROP ( Tieleman and Hinton , 2012 ) can approximately simulate MOMENTUM ( Polyak , 1964 ) if the term in the denominator of their parameter updates is allowed to grow very large . However , these relationships may not matter in practice . For example , the settings of ADAM ’ s hyperparameters that allow it to match the performance of MOMENTUM may be too difficult to find ( for instance , they may be infinite ) . In this paper , we demonstrate two important and interrelated points about empirical comparisons of neural network optimizers . First , we show that inclusion relationships between optimizers actually matter in practice ; in our experiments , more general optimizers never underperform special cases . Despite conventional wisdom ( Wilson et al. , 2017 ; Balles and Hennig , 2017 ) , we find that when carefully tuned , ADAM and other adaptive gradient methods never underperform MOMENTUM or SGD . Second , we demonstrate the sensitivity of optimizer comparisons to the hyperparameter tuning protocol . By comparing to previous experimental evaluations , we show how easy it is to change optimizer rankings on a given workload ( model and dataset pair ) by changing the hyperparameter tuning protocol , with optimizer rankings stabilizing according to inclusion relationships as we spend more and more effort tuning . Our findings raise serious questions about the practical relevance of conclusions drawn from these sorts of empirical comparisons . The remainder of this paper is structured as follows . In Section 2 , we review related work , focusing on papers that make explicit claims about optimizer comparisons in deep learning and application papers that provide evidence about the tuning protocols of practitioners . We develop our definition of first-order optimizers in Section 3 along with a notion of inclusion relationships between optimizers . We present our experimental results in Section 4 . Despite thorny methodological issues over how to avoid biases in comparisons due to search spaces that favor one optimizer over another , we believe that our experimental methodology is an acceptable compromise and has substantial practical relevance . Among other results , we show that the inclusion hierarchy of update rules is almost entirely predictive of optimizer comparisons . In particular , NADAM ( Dozat , 2016 ) achieves the best top-1 validation accuracy on ResNet-50 on ImageNet in our experiments . The 77.1 % we obtain with NADAM , although not as good as the 77.6 % obtained using learned data augmentation by Cubuk et al . ( 2018 ) , is better than the best existing published results using any of the more standard pre-processing pipelines ( 76.5 % , due to Goyal et al . ( 2017 ) using MOMENTUM ) . 2 BACKGROUND AND RELATED WORK . Our work was inspired by the recent studies of neural network optimizers by Wilson et al . ( 2017 ) and Schneider et al . ( 2019 ) . Wilson et al . ( 2017 ) constructed a simple classification problem in which adaptive gradient methods ( e.g . ADAM ) converge to provably worse solutions than standard gradient methods . However , crucially , their analysis ignored the parameter in the denominator of some adaptive gradient methods . Wilson et al . ( 2017 ) also presented experiments in which ADAM produced worse validation accuracy than SGD across all deep learning workloads considered . However , they only tuned over the learning rate and learning rate decay scheme in their experiments , leaving all other parameters of ADAM at fixed default values . Despite these findings , adaptive gradient methods continue to be popular since the work of Wilson et al . ( 2017 ) . Schneider et al . ( 2019 ) presented a benchmark suite ( DEEPOBS ) for deep learning optimizers and reported that there was no single best optimizer across the workloads they considered . Yet Schneider et al . ( 2019 ) only tuned the learning rate of each optimizer and left all other hyperparameters at some fixed default values . As we discuss in Section 4.3 , the choices of hyperparameter tuning protocols in Wilson et al . ( 2017 ) and Schneider et al . ( 2019 ) may be the most important factor preventing their results from being relevant to practical choices about which optimizer to use . Hyperparameter tuning is a crucial step of the deep learning pipeline ( Bergstra and Bengio , 2012 ; Snoek et al. , 2012 ; Sutskever et al. , 2013 ; Smith , 2018 ) , so it is critical for papers studying optimizers to match as closely as possible the tuning protocols of an ideal practitioner . Yet , tuning protocols often differ between works studying neural network optimizers and works concerned with training neural networks to solve specific problems . Recent papers that study or introduce optimization algorithms tend to compare to ADAM and RMSPROP without tuning their respective hyperparameters , presumably to simplify their experiments . It is standard to leave at the common default value of 10−8 for ADAM and 10−10 for RMSPROP ( Tieleman and Hinton , 2012 ; Kingma and Ba , 2015 ; Dozat , 2016 ; Balles and Hennig , 2017 ; Loshchilov and Hutter , 2017 ; Zou and Shen , 2018 ; Ma and Yarats , 2018 ; Bernstein et al. , 2018 ; Chen et al. , 2019 ; Zou et al. , 2019 ) . Others do not even report the value of used ( Balles and Hennig , 2017 ; Zhang and Mitliagkas , 2017 ; Keskar and Socher , 2017 ; Chen et al. , 2018 ; Zhou et al. , 2018 ; Aitchison , 2018 ; Reddi et al. , 2019 ; Luo et al. , 2019 ) . There are exceptions . Zaheer et al . ( 2018 ) and Liu et al . ( 2019 ) considered values orders of magnitude larger than the standard default . However , the experiments in both papers gave only a limited consideration to , testing at most two values while tuning ADAM . De et al . ( 2018 ) is the only work we found that considered a broad range of values for . Both Zaheer et al . ( 2018 ) and De et al . ( 2018 ) found that non-default values of outperformed the default . While it is also extremely common in applications to use a default value of , some notable papers tuned and selected values up to eight orders of magnitude away from the common defaults . Szegedy et al . ( 2016 ) used = 1 for RMSPROP ; Liu et al . ( 2019 ) reported that their results were sensitive to and set = 10−6 for ADAM ; Tan et al . ( 2019 ) and Tan and Le ( 2019 ) set = 10−3 for RMSPROP , the latter achieving state-of-the-art ImageNet top-1 accuracy . In reinforcement learning , Hessel et al . ( 2017 ) set = 1.5× 10−4 . Despite being introduced solely to prevent division by zero1 , ADAM ’ s can be interpreted in ways that suggest the optimal choice is problem-dependent . If ADAM is interpreted as an empirical , diagonal approximation to natural gradient descent ( Kingma and Ba , 2015 ) , can be viewed as a multi-purpose damping term whose role is to improve the conditioning of the Fisher , in analogy to the approximate second-order method considered by Becker and Le Cun ( 1988 ) . We can also view as setting a trust region radius ( Martens and Grosse , 2015 ; Adolphs et al. , 2019 ) and controlling an interpolation between momentum and diagonal natural gradient descent , by either diminishing or increasing the effect of vt on the update direction . Under either interpretation , the best value for will be problem-dependent and likely benefit from tuning . 3 WHAT IS AN OPTIMIZER ? . Optimization algorithms are typically defined by their update rule , which is controlled by hyperparameters that determine its behavior ( e.g . the learning rate ) . Consider a differentiable loss function ` : Rd → R whose vector of first partial derivatives is given by∇ ` ( θ ) ( more generally , ∇ ` ( θ ) might be a stochastic estimate of the true gradient ) . In our context , ` represents the loss function computed over an entire dataset by a neural network and θ ∈ Rd represents the vector of model parameters . The optimization problem is to find a point that ( at least locally ) minimizes ` . First-order iterative methods for this problem ( Nesterov , 2018 ) construct a sequence θt of iterates converging to a local minimum θ ? using queries to ` and∇ ` . The sequence θt is constructed by an update ruleM , which determines the next iterate θt+1 from the history Ht of previous iterates along with their function and gradient values , Ht = { θs , ∇ ` ( θs ) , ` ( θs ) } ts=0 , and a setting of hyperparameters φ : N → Rn . Given an initial parameter value θ0 ∈ Rd , the sequence of points visited by an optimizer with update ruleM is given by , θt+1 =M ( Ht , φt ) . ( 1 ) The stochastic gradient descent algorithm ( SGD ; Robbins and Monro , 1951 ) is one of the simplest such methods used for training neural networks . SGD is initialized with θ0 ∈ Rd , and its hyperparameter is a learning rate schedule η : N → ( 0 , ∞ ) . The SGD update rule is given by SGD ( Ht , ηt ) = θt − ηt∇ ` ( θt ) . The MOMENTUM method due to Polyak ( 1964 ) generalizes the SGD method by linearly combining the gradient direction with a constant multiple of the previous parameter update . Its hyperparameters are a learning rate schedule η : N → ( 0 , ∞ ) and a momentum parameter γ ∈ [ 0 , ∞ ) , MOMENTUM ( Ht , ηt , γ ) = θt − ηt∇ ` ( θt ) + γ ( θt − θt−1 ) . ( 2 ) There has been an explosion of novel first-order methods in deep learning , all of which fall into this standard first-order scheme . In Table 1 we list the first-order update rules considered in this paper . The difference between optimizers is entirely captured by the choice of update ruleM and hyperparameters φ . Since the roles of optimizer hyperparameters on neural network loss functions are not well-understood , most practitioners tune a subset of the hyperparameters to maximize performance over a validation set , while leaving some hyperparameters at fixed default values . The choice of which hyperparameters to tune determines an effective family of update rules , and this family is the critical object from a practitioners perspective . Thus , in analogy to ( overloaded ) function declarations in C++ , we define an optimizer by an update rule “ signature , ” the update rule name together with the free hyperparameter arguments . For example , in this definition MOMENTUM ( · , ηt , γ ) is not the same optimizer as MOMENTUM ( · , ηt , 0.9 ) , because the latter has two free hyperparameters while the former only has one . ADAM with the default is “ different ” from ADAM with tuned .

Summary: This paper presents experimental data supporting the claim that the under aggressive hyper-parameter tuning different optimizers are essentially ranked by inclusion --- if the hyper-parameters of method A can simulate any setting of the hyper-parameters of method B then under aggressive hyper-parameter tuning A will dominate B.  One way to achieve this rather trivially is to do a hyper-parameter search for B and then set the hyper-parameters of A so that A is simulating B.  But the point here is that direct and feasible tuning of A with dominate B even in the case where A has more hyper-parameters and where hyper-parameter optimization of A would seem to be more difficult.  An important conclusion is that without loss of generality one can always use Adam even in vision where SGD is currently the dominant optimizer used in practice.  Another important conclusion is that quasi-random hyper-parameter optimization is quite effective.