Year: 2019

Abstract:
Selecting an optimizer is a central step in the contemporary deep learning pipeline. In this paper, we demonstrate the sensitivity of optimizer comparisons to the hyperparameter tuning protocol. Our findings suggest that the hyperparameter search space may be the single most important factor explaining the rankings obtained by recent empirical comparisons in the literature. In fact, we show that these results can be contradicted when hyperparameter search spaces are changed. As tuning effort grows without bound, more general optimizers should never underperform the ones they can approximate (i.e., Adam should never perform worse than momentum), but recent attempts to compare optimizers either assume these inclusion relationships are not practically relevant or restrict the hyperparameters in ways that break the inclusions. In our experiments, we find that inclusion relationships between optimizers matter in practice and always predict optimizer comparisons. In particular, we find that the popular adaptive gradient methods never underperform momentum or gradient descent. We also report practical tips around tuning often ignored hyperparameters of adaptive gradient methods and raise concerns about fairly benchmarking optimizers for neural network training.

heading: 1 INTRODUCTION
text: The optimization algorithm chosen by a deep learning practitioner determines the training speed and the final predictive performance of their model. To date, there is no theory that adequately explains how to make this choice. Instead, our community relies on empirical studies (Wilson et al., 2017) and benchmarking (Schneider et al., 2019). Indeed, it is the de facto standard that papers introducing new optimizers report extensive comparisons across a large number of workloads. Therefore, to maximize scientific progress, we must have confidence in our ability to make empirical comparisons between optimization algorithms.
Although there is no theory guiding us when comparing optimizers, the popular first-order optimizers form a natural inclusion hierarchy. For example, ADAM (Kingma and Ba, 2015) and RMSPROP (Tieleman and Hinton, 2012) can approximately simulate MOMENTUM (Polyak, 1964) if the term in the denominator of their parameter updates is allowed to grow very large. However, these relationships may not matter in practice. For example, the settings of ADAM’s hyperparameters that allow it to match the performance of MOMENTUM may be too difficult to find (for instance, they may be infinite).
In this paper, we demonstrate two important and interrelated points about empirical comparisons of neural network optimizers. First, we show that inclusion relationships between optimizers actually matter in practice; in our experiments, more general optimizers never underperform special cases. Despite conventional wisdom (Wilson et al., 2017; Balles and Hennig, 2017), we find that when carefully tuned, ADAM and other adaptive gradient methods never underperform MOMENTUM or SGD. Second, we demonstrate the sensitivity of optimizer comparisons to the hyperparameter tuning protocol. By comparing to previous experimental evaluations, we show how easy it is to change optimizer rankings on a given workload (model and dataset pair) by changing the hyperparameter tuning protocol, with optimizer rankings stabilizing according to inclusion relationships as we spend more and more effort tuning. Our findings raise serious questions about the practical relevance of conclusions drawn from these sorts of empirical comparisons.
The remainder of this paper is structured as follows. In Section 2, we review related work, focusing on papers that make explicit claims about optimizer comparisons in deep learning and application papers that provide evidence about the tuning protocols of practitioners. We develop our definition of first-order optimizers in Section 3 along with a notion of inclusion relationships between optimizers. We present our experimental results in Section 4. Despite thorny methodological issues over how to avoid biases in comparisons due to search spaces that favor one optimizer over another, we believe that our experimental methodology is an acceptable compromise and has substantial practical relevance. Among other results, we show that the inclusion hierarchy of update rules is almost entirely predictive of optimizer comparisons. In particular, NADAM (Dozat, 2016) achieves the best top-1 validation accuracy on ResNet-50 on ImageNet in our experiments. The 77.1% we obtain with NADAM, although not as good as the 77.6% obtained using learned data augmentation by Cubuk et al. (2018), is better than the best existing published results using any of the more standard pre-processing pipelines (76.5%, due to Goyal et al. (2017) using MOMENTUM).

heading: 2 BACKGROUND AND RELATED WORK
text: Our work was inspired by the recent studies of neural network optimizers by Wilson et al. (2017) and Schneider et al. (2019). Wilson et al. (2017) constructed a simple classification problem in which adaptive gradient methods (e.g. ADAM) converge to provably worse solutions than standard gradient methods. However, crucially, their analysis ignored the parameter in the denominator of some adaptive gradient methods. Wilson et al. (2017) also presented experiments in which ADAM produced worse validation accuracy than SGD across all deep learning workloads considered. However, they only tuned over the learning rate and learning rate decay scheme in their experiments, leaving all other parameters of ADAM at fixed default values. Despite these findings, adaptive gradient methods continue to be popular since the work of Wilson et al. (2017). Schneider et al. (2019) presented a benchmark suite (DEEPOBS) for deep learning optimizers and reported that there was no single best optimizer across the workloads they considered. Yet Schneider et al. (2019) only tuned the learning rate of each optimizer and left all other hyperparameters at some fixed default values.
As we discuss in Section 4.3, the choices of hyperparameter tuning protocols in Wilson et al. (2017) and Schneider et al. (2019) may be the most important factor preventing their results from being relevant to practical choices about which optimizer to use. Hyperparameter tuning is a crucial step of the deep learning pipeline (Bergstra and Bengio, 2012; Snoek et al., 2012; Sutskever et al., 2013; Smith, 2018), so it is critical for papers studying optimizers to match as closely as possible the tuning protocols of an ideal practitioner. Yet, tuning protocols often differ between works studying neural network optimizers and works concerned with training neural networks to solve specific problems.
Recent papers that study or introduce optimization algorithms tend to compare to ADAM and RMSPROP without tuning their respective hyperparameters, presumably to simplify their experiments. It is standard to leave at the common default value of 10−8 for ADAM and 10−10 for RMSPROP (Tieleman and Hinton, 2012; Kingma and Ba, 2015; Dozat, 2016; Balles and Hennig, 2017; Loshchilov and Hutter, 2017; Zou and Shen, 2018; Ma and Yarats, 2018; Bernstein et al., 2018; Chen et al., 2019; Zou et al., 2019). Others do not even report the value of used (Balles and Hennig, 2017; Zhang and Mitliagkas, 2017; Keskar and Socher, 2017; Chen et al., 2018; Zhou et al., 2018; Aitchison, 2018; Reddi et al., 2019; Luo et al., 2019). There are exceptions. Zaheer et al. (2018) and Liu et al. (2019) considered values orders of magnitude larger than the standard default. However, the experiments in both papers gave only a limited consideration to , testing at most two values while tuning ADAM. De et al. (2018) is the only work we found that considered a broad range of values for . Both Zaheer et al. (2018) and De et al. (2018) found that non-default values of outperformed the default.
While it is also extremely common in applications to use a default value of , some notable papers tuned and selected values up to eight orders of magnitude away from the common defaults. Szegedy et al. (2016) used = 1 for RMSPROP; Liu et al. (2019) reported that their results were sensitive to and set = 10−6 for ADAM; Tan et al. (2019) and Tan and Le (2019) set = 10−3 for RMSPROP, the latter achieving state-of-the-art ImageNet top-1 accuracy. In reinforcement learning, Hessel et al. (2017) set = 1.5× 10−4.
Despite being introduced solely to prevent division by zero1 , ADAM’s can be interpreted in ways that suggest the optimal choice is problem-dependent. If ADAM is interpreted as an empirical, diagonal approximation to natural gradient descent (Kingma and Ba, 2015), can be viewed as a multi-purpose damping term whose role is to improve the conditioning of the Fisher, in analogy to the approximate second-order method considered by Becker and Le Cun (1988). We can also view as setting a trust region radius (Martens and Grosse, 2015; Adolphs et al., 2019) and controlling an interpolation between momentum and diagonal natural gradient descent, by either diminishing or increasing the effect of vt on the update direction. Under either interpretation, the best value for will be problem-dependent and likely benefit from tuning.

heading: 3 WHAT IS AN OPTIMIZER?
text: Optimization algorithms are typically defined by their update rule, which is controlled by hyperparameters that determine its behavior (e.g. the learning rate). Consider a differentiable loss function ` : Rd → R whose vector of first partial derivatives is given by∇`(θ) (more generally,∇`(θ) might be a stochastic estimate of the true gradient). In our context, ` represents the loss function computed over an entire dataset by a neural network and θ ∈ Rd represents the vector of model parameters. The optimization problem is to find a point that (at least locally) minimizes `. First-order iterative methods for this problem (Nesterov, 2018) construct a sequence θt of iterates converging to a local minimum θ? using queries to ` and∇`. The sequence θt is constructed by an update ruleM, which determines the next iterate θt+1 from the history Ht of previous iterates along with their function and gradient values, Ht = {θs,∇`(θs), `(θs)}ts=0, and a setting of hyperparameters φ : N → Rn. Given an initial parameter value θ0 ∈ Rd, the sequence of points visited by an optimizer with update ruleM is given by,
θt+1 =M(Ht, φt). (1) The stochastic gradient descent algorithm (SGD; Robbins and Monro, 1951) is one of the simplest such methods used for training neural networks. SGD is initialized with θ0 ∈ Rd, and its hyperparameter is a learning rate schedule η : N → (0,∞). The SGD update rule is given by SGD(Ht, ηt) = θt − ηt∇`(θt). The MOMENTUM method due to Polyak (1964) generalizes the SGD method by linearly combining the gradient direction with a constant multiple of the previous parameter update. Its hyperparameters are a learning rate schedule η : N → (0,∞) and a momentum parameter γ ∈ [0,∞),
MOMENTUM(Ht, ηt, γ) = θt − ηt∇`(θt) + γ(θt − θt−1). (2) There has been an explosion of novel first-order methods in deep learning, all of which fall into this standard first-order scheme. In Table 1 we list the first-order update rules considered in this paper.
The difference between optimizers is entirely captured by the choice of update ruleM and hyperparameters φ. Since the roles of optimizer hyperparameters on neural network loss functions are not well-understood, most practitioners tune a subset of the hyperparameters to maximize performance over a validation set, while leaving some hyperparameters at fixed default values. The choice of which hyperparameters to tune determines an effective family of update rules, and this family is the critical object from a practitioners perspective. Thus, in analogy to (overloaded) function declarations in C++, we define an optimizer by an update rule “signature,” the update rule name together with the free hyperparameter arguments. For example, in this definition MOMENTUM(·, ηt, γ) is not the same optimizer as MOMENTUM(·, ηt, 0.9), because the latter has two free hyperparameters while the former only has one. ADAM with the default is “different” from ADAM with tuned .

heading: 3.1 THE TAXONOMY OF FIRST-ORDER METHODS
text: The basic observation of this section is that some optimizers can approximately simulate others (i.e., optimizer A might be able to approximately simulate the trajectory of optimizer B for any particular setting of B’s hyperparameters). This is important knowledge because, as a hyperparameter tuning protocol approaches optimality, a more expressive optimizer can never underperform any of its specializations. To capture the concept of one optimizer approximating another, we define the following inclusion relationship between optimizers.
1TensorFlow currently refers to as “a small constant for numerical stability”; https://www. tensorflow.org/versions/r1.15/api_docs/python/tf/train/AdamOptimizer.
Definition 1 (Inclusion relationship). LetM,N be update rules for use in a first-order optimization method. M is a subset or specialization of N , if for all φ : N → Rn, there exists a sequence ψi : N→ Rm, such that for all t ∈ [0,∞) and histories Ht,
lim i→∞
N (Ht, ψit) =M(Ht, φt)
This is denotedM⊆ N , with equalityM = N iffM⊆ N and N ⊆M.
Evidently SGD ⊆ MOMENTUM, since SGD(Ht, ηt) = MOMENTUM(Ht, ηt, 0). Many wellknown optimizers fall naturally into this taxonomy. In particular, we consider RMSPROP with momentum (Tieleman and Hinton, 2012), ADAM (Kingma and Ba, 2015) and NADAM (Dozat, 2016) (see Table 1) and show the following inclusions in the appendix.

heading: SGD ⊆ MOMENTUM ⊆ RMSPROP
text: SGD ⊆ MOMENTUM ⊆ ADAM SGD ⊆ NESTEROV ⊆ NADAM
(3)
Note, some of these inclusions make use of the flexibility of hyperparameter schedules (dependence of ψi on t). In particular, to approximate MOMENTUM with ADAM, one needs to choose a learning rate schedule that accounts for ADAM’s bias correction.
If two optimizers have an inclusion relationship, the more general optimizer can never be worse with respect to any metric of interest, provided the hyperparameters are sufficiently tuned to optimize that metric. Optimally-tuned MOMENTUM cannot underperform optimally-tuned SGD, because setting γ = 0 in MOMENTUM recovers SGD. However, optimizers with more hyperparameters might be more expensive to tune, so we should have a theoretical or experimental reason for using (or creating) a more general optimizer. For example, MOMENTUM improves local convergence rates over SGD on twice-differentiable functions that are smooth and strongly convex (Polyak, 1964), and
NESTEROV has globally optimal convergence rates within the class of smooth and strongly convex functions (Nesterov, 1983; 2018).
At first glance, the taxonomy of optimizer inclusions appears to resolve many optimizer comparison questions. However, for a deep learning practitioner, there is no guarantee that the inclusion hierarchy is at all meaningful in practice. For example, the hyperparameters that allow ADAM to match or outperform MOMENTUM might not be easily accessible. They might exist only in the limit of very large values, or be so difficult to find that only practitioners with huge computational budgets can hope to discover them. Indeed, empirical studies and conventional wisdom hold that the inclusion hierarchy does not predict optimizer performance for many practical workloads (Wilson et al., 2017; Balles and Hennig, 2017; Schneider et al., 2019). Either these experimental investigations are too limited or the taxonomy of this section is of limited practical interest and provides no guidance about which optimizer to use on a real workload. In the following section we attempt to answer this question experimentally, and show that these inclusion relationships are meaningful in practice.

heading: 4 EXPERIMENTS
text: An empirical comparison of optimizers should aim to inform a careful practitioner. Accordingly, we model our protocol on a practitioner that is allowed to vary all optimization hyperparameters for each optimizer (e.g. αt, β1, β2, for ADAM) in addition to a parameterized learning rate decay schedule, in contrast to studies that fix a subset of the optimization hyperparameters to their default values (e.g. Wilson et al., 2017; Schneider et al., 2019). There is no standard method for selecting the values of these hyperparameters, but most practitioners tune at least a subset of the optimization hyperparameters by running a set of trials to maximize performance over the validation set. In our experiments, we run tens to hundreds of individual trials per workload. Given the variety of workloads we consider, this trial budget covers a wide range of computational budgets.
Selecting the hyperparameter search space for each optimizer is a key methodological choice for any empirical comparison of optimizers. Prior studies have attempted to treat each optimizer fairly by using the same search space for all optimizers (e.g. Wilson et al., 2017; Schneider et al., 2019). However, this requires the assumption that similarly-named hyperparameters should take similar values between optimizers, which is not always true. For example, MOMENTUM and NESTEROV both have similar-looking momentum and learning rate hyperparameters, but NESTEROV tolerates larger values of its momentum hyperparameter (Sutskever et al., 2013), so any fixed search space will likely be more favorable for one of the two. The situation worsens with less closely related optimizers, and designing a search space that is equally appropriate for optimizers with incommensurate hyperparameters is almost impossible. Despite coming with its own set of challenges, it is most informative to compare optimizers assuming the practitioner is allowed to tune hyperparameters for different optimizers independently by way of optimizer-specific search spaces.
In our experiments, we chose the search space for each optimizer by running an initial set of experiments over a relatively large search space. In a typical case, we ran a single set of initial trials per optimizer to select the final search space. However, in some cases we chose the initial search space poorly, so we ran another set of experiments to select the final search space. The effort required to choose each search space cannot simply be quantified by the number of initial trials; the provenance of each search space is difficult to trace exactly. In some cases, our search spaces were informed by published results or prior experience with particular models and optimizers. We note that this is true of all search spaces in the literature: they are hard-won treasures that tend to be refined over many experiments and across many workloads, representing the sum total of our community’s experience. We validated our search spaces by checking that that the optimal hyperparameter values were away from the search space boundaries for all optimizers in all experiments (see Figure 5 in Appendix E). We provide our final search spaces for all experiments in Appendix D. The fact that our final error rates compare favorably to prior published results – including reaching state-of-the-art for our particular configuration of ResNet-50 on ImageNet (see Section 4.2) – supports our claim that our methodology is highly competitive with expert tuning procedures.

heading: 4.1 OVERVIEW OF WORKLOADS AND EXPERIMENTAL DETAILS
text: We investigated the relative performance of optimizers across a variety of image classification and language modeling tasks. For image classification, we trained a simple convolutional neural network (Simple CNN) on Fashion MNIST (Xiao et al., 2017); ResNet-32 (He et al., 2016a) on CIFAR-10 (Krizhevsky, 2009); a CNN on CIFAR-100; VGG-16 (Simonyan and Zisserman, 2014) on CIFAR10; and ResNet-50 on ImageNet (Russakovsky et al., 2015). For language modeling, we trained a 2-layer LSTM model (Hochreiter and Schmidhuber, 1997) on Tolstoy’s War and Peace; and Transformer (Vaswani et al., 2017) on LM1B (Chelba et al., 2014). We used a linear learning rate decay schedule parameterized the same way as Shallue et al. (2019) for all workloads. We used a fixed batch size and a fixed budget of training steps for each workload independent of the optimizer. Table 2 summarizes these workloads and Appendix B provides the full details.
Given a hypercube-shaped search space, our tuning protocol sought to model a practitioner with a fixed budget of trials trying to achieve the best outcome using tens of feasible trials (10, 50, or 100 depending on the workload).2 A feasible trial is any trial that achieves finite training loss. We used quasi-random uniform search (Bousquet et al., 2017), and continued the search until we obtained a fixed number of feasible trials. From those trials we considered two statistics. The first, in order to characterize the best outcome, is a metric of interest (e.g. test accuracy) corresponding to the trial achieving the optimum of some other metric (e.g. validation accuracy). The second, in order to characterize the speed of training, is the number of steps required to reach a fixed validation target conditional on at least one trial in the search having reached that target. We chose the target for each workload based on initial experiments and known values from the literature (see Table 2). We estimated means and uncertainties using the bootstrap procedure described in Appendix C.

heading: 4.2 INCLUSION RELATIONSHIPS MATTER IN PRACTICE
text: Figure 1 shows the final predictive performance of six optimizers on four different workloads after tuning hyperparameters to minimize validation error. Regardless of whether we compare final validation error or test error, the inclusion relationships hold in all cases – a more general optimizer never underperforms any of its specializations within the error bars. Similar results hold for training error (see Figure 9 in Appendix E). Training speed is also an important consideration, and Figure 2 demonstrates that the inclusion relationships also hold within error bars when we compare the number of steps required to reach a target validation error. Moreover, these results confirming the relevance of optimizer inclusion relationships do not depend on the exact step budgets or error targets we chose (see Figure 10 in Appendix E), although large changes to these values would require new experiments.
2Although we used a budget of tens of independent tuning trials throughout this section, in retrospect the best validation error across tuning trials converged quite quickly for our final search spaces, producing good results with fewer than 20 trials in many cases. See Figures 6– 8 in Appendix E.
Of course, just because a more general optimizer is no worse than any of its specializations doesn’t mean the choice of optimizer makes a large difference on all workloads. For some workloads in Figures 1 and 2, all optimizers perform about the same, while other workloads have a clear ranking or even dramatic differences. For example, the choice of optimizer seems to make little difference for ResNet-32 on CIFAR-10; all optimizers achieve similar predictive performance and training speed. On the other hand, Transformer on LM1B exhibits a clear ranking in terms of predictive performance and training speed. For this workload, ADAM needs roughly half the steps that MOMENTUM requires to reach our target error, and, although not shown in Figure 2, roughly six times fewer steps to get the same result as SGD. These differences are clearly significant enough to matter to a practitioner, and highlight the practical importance of choosing the right optimizer for some workloads.
The most general optimizers we considered were RMSPROP, ADAM, and NADAM, which do not include each other as special cases, and whose relative performance is not predicted by inclusion relationships. Across the workloads we considered, none of these optimizers emerged as the clear winner, although ADAM and NADAM generally seemed to have an edge over RMSPROP. For all of these optimizers, we sometimes had to set the parameter orders of magnitude larger than the default value in order to get good results. In particular, we achieved a validation accuracy of 77.1% for ResNet-50 on ImageNet using NADAM with = 9475, a result that exceeds the 76.5% achieved by Goyal et al. (2017) using MOMENTUM. Across just these 4 workloads, the range of the optimal values of the parameter spanned 10 orders of magnitude. Faced with this reality, a practitioner might reasonably doubt their ability to find a value near the optimum. However, we found that we could reasonably expect to find a suitable value with only tens of trials. When tuning for ADAM or NADAM over a large range, we found it more efficient to search over ( , α0/ ) instead of ( , α0); see Appendix D for more details.

heading: 4.3 RECONCILING DISAGREEMENTS WITH PREVIOUS WORK
text: In order to confirm that differences in hyperparameter tuning protocols explain the differences between our conclusions and those of Wilson et al. (2017) and Schneider et al. (2019), we reproduced a representative subset of their results and then inverted, or at least collapsed, the ranking over optimizers just by expanding the hyperparameter search space.
The left pane of Figure 3 shows our experiments on VGG on CIFAR-10 using code released by Wilson et al. (2017). When we match their protocol and perform their grid search over the initial learning rate and no other tuning, we reproduce their original result showing worse test error for RMSPROP and ADAM. However, when we tune the momentum parameter and with random search, all four optimizers reach nearly identical test error rates.3 With our learning rate schedule search space, merely tuning the learning rate schedule was enough to make all optimizers reach the same test error within error bars. When we additionally tuned the optimization hyperparameters and weight decay in our setup we also get similar results for all optimizers, removing any evidence the inclusion relationships might be violated in practice.
Figure 4 shows our results with different tuning protocols for a CNN on CIFAR-100 and an LSTM language model trained on War and Peace to match the experiments in Schneider et al. (2019). As reported by Schneider et al. (2019), if we only tune the learning rate without tuning the decay schedule or other optimizer hyperparameters, ADAM does worse than MOMENTUM for the CNN and SGD performs slightly better than ADAM and MOMENTUM on the War and Peace dataset, although Schneider et al. (2019) found a larger advantage for SGD. However, once we tune the all the optimizer hyperparameters, ADAM does better than MOMENTUM which does better than SGD, as predicted by the inclusion relationships.
We conclude that the reason both Schneider et al. (2019) and Wilson et al. (2017) observed a ranking that, at first glance, contradicts the inclusion relationships is because they were not tuning enough of the hyperparameters. If we recast their results in our terminology where ADAM with default is a different optimizer than ADAM with tuned then there is no contradiction with our results and it becomes clear immediately that they do not consider the most interesting form of ADAM for practitioners.
3Wilson et al. (2017) selected trials to minimize the training loss and then report test set results. As Figure 3 shows, removing this somewhat non-standard choice and tuning on a validation set and reporting test set results does not change anything.

heading: 5 CONCLUSIONS
text: Inspired by the recent efforts of Wilson et al. (2017) and Schneider et al. (2019), we set out to provide a detailed empirical characterization of the optimizer selection process in deep learning. Our central finding is that inclusion relationships between optimizers are meaningful in practice. When tuning all available hyperparameters under a realistic protocol at scales common in deep learning, we find that more general optimizers never underperform their special cases. In particular, we found that RMSPROP, ADAM, and NADAM never underperformed SGD, NESTEROV, or MOMENTUM under our most exhaustive tuning protocol. We did not find consistent trends when comparing optimizers that could not approximate each other. We also found workloads for which there was not a statistically significant separation in the optimizer ranking.
Our experiments have some important limitations and we should be careful not to overgeneralize from our results. The first major caveat is that we did not measure the effects of varying the batch size. Recent empirical work (Shallue et al., 2019; Zhang et al., 2019) has shown that increasing the batch size can increase the gaps between training times for different optimizers, with the gap from SGD to MOMENTUM (Shallue et al., 2019) and from MOMENTUM to ADAM (Zhang et al., 2019) increasing with the batch size. Nevertheless, we strongly suspect that the inclusion relations would be predictive at any batch size under a tuning protocol similar to the one we used. The second important caveat of our results is that they inevitably depend on the tuning protocol and workloads that we considered. Although we made every attempt to conduct realistic experiments, we should only expect our detailed findings to hold for similar workloads under similar protocols, namely uniform quasi-random tuning for tens to hundreds of trials, over hypercube search spaces, and with our specific learning rate schedule parameterization. Nevertheless, these caveats reinforce our central point: all empirical comparisons of neural network optimizers depend heavily on the hyperparameter tuning protocol, perhaps far more than we are used to with comparisons between model architectures.
If we were to extract “best practices” from our findings, then we suggest the following. If we can afford tens or more runs of our code, we should tune all of the hyperparameters of the popular adaptive gradient methods. Just because two hyperparameters have a similar role in two different update rules doesn’t mean they should take similar values— optimization hyperparameters tend to be coupled and the optimal value for one may depend on how the others are set. Our results also confirm that the optimal value of Adam’s is problem-dependent, so the onus is on empirical studies that fix = 10−8 to defend that choice. Finally, we should be skeptical of empirical comparisons of optimizers in papers, especially if an optimizer underperforms any of its specializations. When we do inevitably compare optimizers, we should report search spaces and highlight decisions about what hyperparameters were tuned when interpreting results.

heading: A OPTIMIZER INCLUSIONS
text: Table 1 summarizes the update rules for the optimizers we consider in this work. We assume update rules as implemented in TensorFlow r1.15. RMSPROP includes momentum. Here we prove the their inclusion relationships, see Definition 1.

heading: MOMENTUM can exactly implement SGD
text: MOMENTUM(It, ηt, 0) = SGD(It, ηt), so SGD ⊆ MOMENTUM.

heading: NESTEROV can exactly implement SGD
text: NESTEROV(It, ηt, 0) = SGD(It, ηt), so SGD ⊆ NESTEROV. RMSPROP with momentum can exactly implement MOMENTUM
Consider RMSPROP(It, ηt, γ, ρ = 1, = 0), so that
mt+1 = γmt + ηt∇`(θt) , θt+1 = θt −mt+1 .
This is equivalent to MOMENTUM, since
m(RMSPROP)t+1 ≡ ηtv (MOMENTUM) t+1 .
Thus RMSPROP(It, ηt, γ, 1, 0) = MOMENTUM(It, ηt, γ), so MOMENTUM ⊆ RMSPROP. RMSTEROV can exactly implement NESTEROV
Consider RMSTEROV(It, ηt, γ, ρ = 1, = 0), so that
mt+1 = γmt + ηt∇`(θt) , θt+1 = θt − [γmt+1 + ηt∇`(θt)] .
This is equivalent to MOMENTUM, since
m(RMSPROP)t+1 ≡ ηtv (NESTEROV) t+1 .
Thus RMSTEROV(It, ηt, γ, 1, 0) = MOMENTUM(It, ηt, γ), so MOMENTUM ⊆ RMSTEROV. ADAM can approximate MOMENTUM for large
Consider ADAM(It, αt = ηt(1− γt), β1 = γ, β2 = 0, ), so that
mt+1 = γmt + (1− γ)∇`(θt) ,
θt+1 = θt − ηt
(1− γ)
[ mt+1
|∇`(θt)|/ + 1
] .
If is large, so that |∇`(θt)|/ 1, then
mt+1 = γmt + (1− γ)∇`(θt) ,
θt+1 = θt − ηt mt+1 1− γ .
This is equivalent to MOMENTUM, since
m(ADAM)t ≡ (1− γ) v (MOMENTUM) t .
Thus lim →∞ ADAM(It, ηt(1−γt), γ, 0, ) = MOMENTUM(It, ηt, γ), so MOMENTUM ⊆ ADAM NADAM can approximate NESTEROV for large
Consider NADAM(It, αt = ηt(1− γt), β1 = γ, β2 = 0, ), so that
mt+1 = γmt + (1− γ)∇`(θt) ,
θt+1 = θt − ηt
(1− γ) [ γmt+1 + (1− γ)∇`(θt) |∇`(θt)|/ + 1 ] .
If is large, so that |∇`(θt)|/ 1, then
mt+1 = γmt + (1− γ)∇`(θt) , θt+1 = θt − ηt [ γmt+1 1− γ +∇`(θt) ] .
This is equivalent to NESTEROV, since
m(NADAM)t ≡ (1− γ) v (NESTEROV) t
Thus lim →∞ NADAM(It, ηt(1 − γt), γ, 0, ) = NESTEROV(It, ηt, γ), so NESTEROV ⊆ NADAM.

heading: B WORKLOAD DETAILS
text: This section details the datasets and models summarized in Table 2.

heading: B.1 DATASET DESCRIPTIONS
text: For Fashion MNIST, CIFAR-10, ImageNet, and LM1B, our setup was identical to Shallue et al. (2019) except for the image pre-processing details described below. For War and Peace, our setup was identical to the “Tolstoi” dataset of Schneider et al. (2019).
CIFAR-10/100: We pre-processed images by subtracting the average value across all pixels and channels and dividing by the standard deviation.4 For experiments with the ResNet-32 and CNN models, we followed the standard data augmentation scheme used in He et al. (2016a): 4 pixels padded on each side with single random crop from padded image or its horizontal reflection. We did not use random cropping for experiments with VGG for consistency with Wilson et al. (2017).
ImageNet: We augmented images at training time by resizing each image, taking a random crop of 224× 224 pixels, randomly horizontally reflecting the cropped images, and randomly distorting the image colors. At evaluation time, we performed a single central crop of 224 × 224 pixels. In both training and evaluation, we then subtracted the global mean RGB value from each pixel using the values computed by Simonyan and Zisserman (2014).5

heading: B.2 MODEL DESCRIPTIONS
text: Simple CNN is identical to the base model described in Shallue et al. (2019). It consists of 2 convolutional layers with max pooling followed by 1 fully connected layer. The convolutional layers use 5 × 5 filters with stride 1, “same” padding, and ReLU activation function. Max pooling uses a 2×2 window with stride 2. Convolutional layers have 32 and 64 filters each and the fully connected layer has 1024 units. It does not use batch normalization.
CNN is the “All-CNN-C” model from Springenberg et al. (2014), as used in Schneider et al. (2019). The model consists of 3 convolutional layer blocks with max pooling. The convolutional layers use 5× 5 filters with stride 1, “same” padding, and ReLU activation function. Max pooling uses a 2× 2 window with stride 2. Convolutional layer blocks have 96, 192 and 192 filters each. As in Schneider et al. (2019), we used L2 regularization of 5× 10−4. ResNet is described in He et al. (2016a). We used the improved residual block described in He et al. (2016b). We used batch normalization (Ioffe and Szegedy, 2015) with exponential moving average (EMA) decay of 0.997 for ResNet-32, and ghost batch normalization (Hoffer et al., 2017) with ghost batch size of 32 and EMA decay of 0.9 for ResNet-50.
VGG is based on “model C” from Simonyan and Zisserman (2014). It consists of 13 convolutional layers followed by 3 fully connected hidden layers. We followed the modification used by Wilson et al. (2017) with batch normalization layers.
4We used the TensorFlow op tf.image.per image standardization. 5See https://gist.github.com/ksimonyan/211839e770f7b538e2d8#description
for the mean RGB values used.
LSTM is a two hidden-layer LSTM model (Hochreiter and Schmidhuber, 1997) identical to the model used in Schneider et al. (2019). It uses 128 embedding dimensions and 128 hidden units.
Transformer is the “base” model described in (Vaswani et al., 2017). We used it as an autoregressive language model by applying the decoder directly to the sequence of word embeddings for each sentence. Unlike the default implementation, we removed dropout regularization and used separate weight matrices for the input embedding layer and the pre-softmax linear transformation, as we observed these choices led to better performing models.

heading: C ESTIMATING TRIAL OUTCOMES VIA BOOTSTRAP
text: Our tuning protocol corresponds to running trials with quasi-random hyperparameter values sampled uniformly from the search space until K feasible trials are obtained, with K depending on the workload. We then select the best trial, based on our statistic of interest, over those K trials.
We used the following bootstrap procedure to estimate means and uncertainties of our tuning protocol. We ran N > K trials, with N depending on the workload. Then, for each bootstrap sample, we resampled the dataset of N trials with replacement and computed our statistic on the first K trials of the resampled dataset. We collected 100 such bootstrap samples each time, and from those computed the means, 5th percentiles, and 95th percentiles of the bootstrap distribution. We used this procedure to generate the means and error bars for each plot.
Simple CNN on Fashion MNIST used (K,N) = (100, 500); ResNet-32 on CIFAR-100 used (K,N) = (100, 500); ResNet-50 on ImageNet used (K,N) = (50, 250); Transformer on LM1B used (K,N) = (50, 100); VGG on CIFAR-10 with our code used (K,N) = (50, 250) for tuning the learning rate schedule and (K,N) = (100, 500) for tuning the learning rate schedule, {γ, β1, β2, }, and L2 regularization; CNN on CIFAR-10 used (K,N) = (100, 500); LSTM on War and Peace used (K,N) = (10, 50) for tuning just the learning rate and (K,N) = (100, 500) for tuning the learning rate schedule and {γ, β1, β2, }. The sole exceptions to this bootstrap procedure are the two left panels of Figure 3, for which we used a similar procedure to Wilson et al. (2017) to ensure comparability. For each optimizer, we selected the trial that minimized validation error in our final search space and ran the same hyperparameter values 5 times, reporting the mean, minimum, and maximum test error over those 5 runs in Figure 3. This is slightly different to Wilson et al. (2017), who chose the trial that minimized training error and reported validation error. When tuning the learning rate and {γ, }, we used 24 trials per optimizer in the initial search space (which we used to select the final search space), and 16 trials per optimizer in the final search space.

heading: D HYPERPARAMETER SEARCH SPACES
text: When tuning hyperparameters over a large range, we found that our search could sometimes be made more efficient if we parametrized the search space in a way that decorrelated the axes of the space. For example, with MOMENTUM and NESTEROV we observed a clear relationship between the initial learning rate η0 and the momentum parameter γ; smaller values of η0 require larger values of γ for good performance, and vice versa. Indeed, Shallue et al. (2019) suggested that these optimizers are governed by the “effective learning rate” ηeff = η0/(1 − γ), and inspired by this, we found that searching over (η0, ηeff) instead of (η0, γ) usually led to a more efficient hyperparameter search. Similarly, with ADAM and NADAM we observed a relationship between the initial learning rate α0 and the parameter; larger values of α0 require larger values of for good performance, and vice versa. This is not surprising given the analysis in Appendix A that showed that, for large , α0/ is analogous to the effective learning rate of ADAM and NADAM. We found that searching over ( , α0/ ) was usually more efficient than searching over ( , α0). We used these techniques in a subset of our experiments.
Below we report the search spaces used for our experiments. We include both the initial search spaces used to refine the search spaces, and the final spaces used to generate the plots. When only one search space was used, we denote the initial space as final. η0, α0, 1 − γ, 1 − β1, 1 − β2, , and combinations thereof are always tuned on a log scale. The number of samples from each search space is specified in Appendix C.

heading: D.1 CNN ON FASHION MNIST
text: We used linear learning rate decay for all experiments. We tuned the number of decay steps within [0.5, 1.0] times the number of training steps and the learning rate decay factor within {10−3, 10−2, 10−1}. We did not use L2 regularization or weight decay.

heading: D.2 RESNET-32 ON CIFAR-10
text: We used linear learning rate decay for all experiments. We tuned the number of decay steps within [0.5, 1.0] times the number of training steps and the learning rate decay factor f within the values shown in the tables below. λL2 denotes the L2 regularization coefficient.

heading: D.3 RESNET-50 ON IMAGENET
text: We used linear learning rate decay for all experiments. We tuned the number of decay steps within [0.5, 1.0] times the number of training steps and the learning rate decay factor f within the values shown in the tables below. λwd denotes the weight decay coefficient and τ denotes the label smoothing coefficient.

heading: D.4 TRANSFORMER ON LM1B
text: We used linear learning rate decay for all experiments. We tuned the number of decay steps within [0.5, 1.0] times the number of training steps and the learning rate decay factor within {10−4, 10−3, 10−2, 10−1, 1}.
D.5 VGG ON CIFAR-10 USING CODE FROM WILSON ET AL. (2017)

heading: D.5.1 GRID SEARCH OVER LEARNING RATE
text: We tuned over the same grid of initial learning rate values for each optimizer as Wilson et al. (2017). As in Wilson et al. (2017), we decayed the initial learning rate by a factor of 0.5 every 25 epochs and used a fixed L2 regularization coefficient of 0.0005.
D.5.2 TUNING LEARNING RATE & {γ, }
We used our quasi-random tuning protocol to tune over the initial learning rate, MOMENTUM’s γ, RMSPROP’s , and ADAM’s . As in Wilson et al. (2017), we decayed the initial learning rate by a factor of 0.5 every 25 epochs and used a fixed L2 regularization coefficient of 0.0005.
D.6 VGG ON CIFAR-10 USING OUR CODE
We used linear learning rate decay for all experiments. We tuned the number of decay steps within [0.5, 1.0] times the number of training steps and the learning rate decay factor within {10−4, 10−3, 10−2, 10−1}.

heading: D.6.1 TUNING LEARNING RATE SCHEDULE
text: We fixed all optimizer hyperparameters excluding the learning rate to match those specified in Wilson et al. (2017). As in Wilson et al. (2017), we used a fixed L2 regularization coefficient of 0.0005.
D.6.2 TUNING LEARNING RATE SCHEDULE & {γ, β1, β2, , λL2}

heading: D.7 CNN ON CIFAR-100

heading: D.7.1 TUNING CONSTANT LEARNING RATE
text: We fixed all optimizer hyperparameters excluding the learning rate to match those specified in Schneider et al. (2019).

heading: D.7.2 TUNING LEARNING RATE SCHEDULE
text: We used linear learning rate decay, and tuned the number of decay steps within [0.5, 1.0] times the number of training steps and the learning rate decay factor within {10−4, 10−3, 10−2, 10−1}.
D.7.3 TUNING CONSTANT LEARNING RATE & {γ, β1, β2, }
For SGD, we reused the results from Appendix D.7.1, since there were no additional hyperparameters to tune.
D.7.4 TUNING LEARNING RATE SCHEDULE & {γ, β1, β2, }
We used linear learning rate decay, and tuned the number of decay steps within [0.5, 1.0] times the number of training steps and the learning rate decay factor within {10−4, 10−3, 10−2, 10−1}. For SGD, we reused the results from Appendix D.7.2, since there were no additional hyperparameters to tune.

heading: D.8 LSTM ON WAR AND PEACE
text: D.8.1 TUNING CONSTANT LEARNING RATE
D.8.2 TUNING LEARNING RATE SCHEDULE & {γ, β1, β2, }
We used linear learning rate decay, and tuned the number of decay steps within [0.5, 1.0] times the number of training steps and the learning rate decay factor f within the values shown in the tables below.

heading: E ADDITIONAL PLOTS

Review summaries:
This paper presents experimental data supporting the claim that the under aggressive hyper-parameter tuning different optimizers are essentially ranked by inclusion --- if the hyper-parameters of method A can simulate any setting of the hyper-parameters of method B then under aggressive hyper-parameter tuning A will dominate B.  One way to achieve this rather trivially is to do a hyper-parameter search for B and then set the hyper-parameters of A so that A is simulating B.  But the point here is that direct and feasible tuning of A with dominate B even in the case where A has more hyper-parameters and where hyper-parameter optimization of A would seem to be more difficult.  An important conclusion is that without loss of generality one can always use Adam even in vision where SGD is currently the dominant optimizer used in practice.  Another important conclusion is that quasi-random hyper-parameter optimization is quite effective.

The paper provides an empirical comparison of a set of first-order optimization methods for deep learning models. Those optimizers include stochastic gradient descent, momentum  method, RMSProp, Adam, Nesterov, and Nadam, which arguably covers all popular variants used in the literature. Although it is not the first empirical study on this topic, its conclusion differs slightly. The conclusion is a rather intuitive one: With proper parameter search, the 'richer', more powerful optimizers tend to work better, regardless of the downstream tasks.