Title: ON EMPIRICAL COMPARISONS OF OPTIMIZERS FOR DEEP LEARNING Anonymous authors Paper under double-blind review

Abstract: Selecting an optimizer is a central step in the contemporary deep learning pipeline. In this paper, we demonstrate the sensitivity of optimizer comparisons to the hyperparameter tuning protocol. Our findings suggest that the hyperparameter search space may be the single most important factor explaining the rankings obtained by recent empirical comparisons in the literature. In fact, we show that these results can be contradicted when hyperparameter search spaces are changed. As tuning effort grows without bound, more general optimizers should never underperform the ones they can approximate (i.e., Adam should never perform worse than momentum), but recent attempts to compare optimizers either assume these inclusion relationships are not practically relevant or restrict the hyperparameters in ways that break the inclusions. In our experiments, we find that inclusion relationships between optimizers matter in practice and always predict optimizer comparisons. In particular, we find that the popular adaptive gradient methods never underperform momentum or gradient descent. We also report practical tips around tuning often ignored hyperparameters of adaptive gradient methods and raise concerns about fairly benchmarking optimizers for neural network training.

1. INTRODUCTION:

The optimization algorithm chosen by a deep learning practitioner determines the training speed and the final predictive performance of their model. To date, there is no theory that adequately explains how to make this choice. Instead, our community relies on empirical studies (Wilson et al., 2017) and benchmarking (Schneider et al., 2019) . Indeed, it is the de facto standard that papers introducing new optimizers report extensive comparisons across a large number of workloads. Therefore, to maximize scientific progress, we must have confidence in our ability to make empirical comparisons between optimization algorithms.

Although there is no theory guiding us when comparing optimizers, the popular first-order optimizers form a natural inclusion hierarchy. For example, ADAM (Kingma and Ba, 2015) and RMSPROP (Tieleman and Hinton, 2012) can approximately simulate MOMENTUM (Polyak, 1964) if the term in the denominator of their parameter updates is allowed to grow very large. However, these relationships may not matter in practice. For example, the settings of ADAM's hyperparameters that allow it to match the performance of MOMENTUM may be too difficult to find (for instance, they may be infinite).

In this paper, we demonstrate two important and interrelated points about empirical comparisons of neural network optimizers. First, we show that inclusion relationships between optimizers actually matter in practice; in our experiments, more general optimizers never underperform special cases. Despite conventional wisdom (Wilson et al., 2017; Balles and Hennig, 2017) , we find that when carefully tuned, ADAM and other adaptive gradient methods never underperform MOMENTUM or SGD. Second, we demonstrate the sensitivity of optimizer comparisons to the hyperparameter tuning protocol. By comparing to previous experimental evaluations, we show how easy it is to change optimizer rankings on a given workload (model and dataset pair) by changing the hyperparameter tuning protocol, with optimizer rankings stabilizing according to inclusion relationships as we spend more and more effort tuning. Our findings raise serious questions about the practical relevance of conclusions drawn from these sorts of empirical comparisons.

The remainder of this paper is structured as follows. In Section 2, we review related work, focusing on papers that make explicit claims about optimizer comparisons in deep learning and application papers that provide evidence about the tuning protocols of practitioners. We develop our definition of first-order optimizers in Section 3 along with a notion of inclusion relationships between optimizers. We present our experimental results in Section 4. Despite thorny methodological issues over how to avoid biases in comparisons due to search spaces that favor one optimizer over another, we believe that our experimental methodology is an acceptable compromise and has substantial practical relevance. Among other results, we show that the inclusion hierarchy of update rules is almost entirely predictive of optimizer comparisons. In particular, NADAM (Dozat, 2016) achieves the best top-1 validation accuracy on ResNet-50 on ImageNet in our experiments. The 77.1% we obtain with NADAM, although not as good as the 77.6% obtained using learned data augmentation by Cubuk et al. (2018) , is better than the best existing published results using any of the more standard pre-processing pipelines (76.5%, due to Goyal et al. (2017) using MOMENTUM).

2. BACKGROUND AND RELATED WORK:

Our work was inspired by the recent studies of neural network optimizers by Wilson et al. (2017) and Schneider et al. (2019) . Wilson et al. (2017) constructed a simple classification problem in which adaptive gradient methods (e.g. ADAM) converge to provably worse solutions than standard gradient methods. However, crucially, their analysis ignored the parameter in the denominator of some adaptive gradient methods. Wilson et al. (2017) also presented experiments in which ADAM produced worse validation accuracy than SGD across all deep learning workloads considered. However, they only tuned over the learning rate and learning rate decay scheme in their experiments, leaving all other parameters of ADAM at fixed default values. Despite these findings, adaptive gradient methods continue to be popular since the work of Wilson et al. (2017) . Schneider et al. (2019) presented a benchmark suite (DEEPOBS) for deep learning optimizers and reported that there was no single best optimizer across the workloads they considered. Yet Schneider et al. (2019) only tuned the learning rate of each optimizer and left all other hyperparameters at some fixed default values.

As we discuss in Section 4.3, the choices of hyperparameter tuning protocols in Wilson et al. (2017) and Schneider et al. (2019) may be the most important factor preventing their results from being relevant to practical choices about which optimizer to use. Hyperparameter tuning is a crucial step of the deep learning pipeline (Bergstra and Bengio, 2012; Snoek et al., 2012; Sutskever et al., 2013; Smith, 2018) , so it is critical for papers studying optimizers to match as closely as possible the tuning protocols of an ideal practitioner. Yet, tuning protocols often differ between works studying neural network optimizers and works concerned with training neural networks to solve specific problems.

Recent papers that study or introduce optimization algorithms tend to compare to ADAM and RMSPROP without tuning their respective hyperparameters, presumably to simplify their experiments. It is standard to leave at the common default value of 10 -8 for ADAM and 10 -10 for RMSPROP (Tieleman and Hinton, 2012; Kingma and Ba, 2015; Dozat, 2016; Balles and Hennig, 2017; Loshchilov and Hutter, 2017; Zou and Shen, 2018; Ma and Yarats, 2018; Bernstein et al., 2018; Chen et al., 2019; Zou et al., 2019) . Others do not even report the value of used (Balles and Hennig, 2017; Zhang and Mitliagkas, 2017; Keskar and Socher, 2017; Chen et al., 2018; Zhou et al., 2018; Aitchison, 2018; Reddi et al., 2019; Luo et al., 2019) . There are exceptions. Zaheer et al. (2018) and Liu et al. (2019) considered values orders of magnitude larger than the standard default. However, the experiments in both papers gave only a limited consideration to , testing at most two values while tuning ADAM. De et al. (2018) is the only work we found that considered a broad range of values for . Both Zaheer et al. (2018) and De et al. (2018) found that non-default values of outperformed the default.

While it is also extremely common in applications to use a default value of , some notable papers tuned and selected values up to eight orders of magnitude away from the common defaults. Szegedy et al. (2016) used = 1 for RMSPROP; Liu et al. (2019) reported that their results were sensitive to and set = 10 -6 for ADAM; Tan et al. (2019) and Tan and Le (2019) set = 10 -3 for RMSPROP, the latter achieving state-of-the-art ImageNet top-1 accuracy. In reinforcement learning, Hessel et al. (2017) 

set = 1.5 × 10 -4 .

Despite being introduced solely to prevent division by zero1 , ADAM's can be interpreted in ways that suggest the optimal choice is problem-dependent. If ADAM is interpreted as an empirical, diagonal approximation to natural gradient descent (Kingma and Ba, 2015) , can be viewed as a multi-purpose damping term whose role is to improve the conditioning of the Fisher, in analogy to the approximate second-order method considered by Becker and Le Cun (1988) . We can also view as setting a trust region radius (Martens and Grosse, 2015; Adolphs et al., 2019) and controlling an interpolation between momentum and diagonal natural gradient descent, by either diminishing or increasing the effect of v t on the update direction. Under either interpretation, the best value for will be problem-dependent and likely benefit from tuning.

3. WHAT IS AN OPTIMIZER?:

Optimization algorithms are typically defined by their update rule, which is controlled by hyperparameters that determine its behavior (e.g. the learning rate). Consider a differentiable loss function : R d → R whose vector of first partial derivatives is given by ∇ (θ) (more generally, ∇ (θ) might be a stochastic estimate of the true gradient). In our context, represents the loss function computed over an entire dataset by a neural network and θ ∈ R d represents the vector of model parameters. The optimization problem is to find a point that (at least locally) minimizes . First-order iterative methods for this problem (Nesterov, 2018) construct a sequence θ t of iterates converging to a local minimum θ using queries to and ∇ . The sequence θ t is constructed by an update rule M, which determines the next iterate θ t+1 from the history H t of previous iterates along with their function and gradient values, H t = {θ s , ∇ (θ s ), (θ s )} t s=0 , and a setting of hyperparameters φ : N → R n . Given an initial parameter value θ 0 ∈ R d , the sequence of points visited by an optimizer with update rule M is given by,

θ t+1 = M(H t , φ t ).

(1) The stochastic gradient descent algorithm (SGD; Robbins and Monro, 1951) is one of the simplest such methods used for training neural networks. SGD is initialized with θ 0 ∈ R d , and its hyperparameter is a learning rate schedule η : N → (0, ∞). The SGD update rule is given by SGD(H t , η t ) = θ t -η t ∇ (θ t ). The MOMENTUM method due to Polyak (1964) generalizes the SGD method by linearly combining the gradient direction with a constant multiple of the previous parameter update. Its hyperparameters are a learning rate schedule η : N → (0, ∞) and a momentum parameter γ ∈ [0, ∞), MOMENTUM(H t , η t , γ) = θ t -η t ∇ (θ t ) + γ(θ t -θ t-1 ).

(2) There has been an explosion of novel first-order methods in deep learning, all of which fall into this standard first-order scheme. In Table 1 we list the first-order update rules considered in this paper.

The difference between optimizers is entirely captured by the choice of update rule M and hyperparameters φ. Since the roles of optimizer hyperparameters on neural network loss functions are not well-understood, most practitioners tune a subset of the hyperparameters to maximize performance over a validation set, while leaving some hyperparameters at fixed default values. The choice of which hyperparameters to tune determines an effective family of update rules, and this family is the critical object from a practitioners perspective. Thus, in analogy to (overloaded) function declarations in C++, we define an optimizer by an update rule "signature," the update rule name together with the free hyperparameter arguments. For example, in this definition MOMENTUM(•, η t , γ) is not the same optimizer as MOMENTUM(•, η t , 0.9), because the latter has two free hyperparameters while the former only has one. ADAM with the default is "different" from ADAM with tuned .

3.1. THE TAXONOMY OF FIRST-ORDER METHODS:

The basic observation of this section is that some optimizers can approximately simulate others (i.e., optimizer A might be able to approximately simulate the trajectory of optimizer B for any particular setting of B's hyperparameters). This is important knowledge because, as a hyperparameter tuning protocol approaches optimality, a more expressive optimizer can never underperform any of its specializations. To capture the concept of one optimizer approximating another, we define the following inclusion relationship between optimizers. Robbins and Monro (1951), MOMENTUM to Polyak (1964) , NESTEROV to Nesterov (1983) , RMSPROP to Tieleman and Hinton (2012), and NADAM to Dozat (2016) . All operations are taken component-wise for vectors. In particular, for x ∈ R d , x 2 is a component-wise power function.

SGD(H t , η t )

θ t+1 = θ t -η t ∇ (θ t ) MOMENTUM(H t , η t , γ) v 0 = 0 v t+1 = γv t + ∇ (θ t ) θ t+1 = θ t -η t v t+1 NESTEROV(H t , η t , γ) v 0 = 0 v t+1 = γv t + ∇ (θ t ) θ t+1 = θ t -η t (γv t+1 + ∇ (θ t )) RMSPROP(H t , η t , γ, ρ, ) v 0 = 1, m 0 = 0 v t+1 = ρv t + (1 -ρ)∇ (θ t ) 2 m t+1 = γm t + η t √ v t+1 + ∇ (θ t ) θ t+1 = θ t -m t+1 ADAM(H t , α t , β 1 , β 2 , ) m 0 = 0, v 0 = 0 m t+1 = β 1 m t + (1 -β 1 )∇ (θ t ) v t+1 = β 2 v t + (1 -β 2 )∇ (θ t ) 2 b t+1 = 1 -β t+1 2 1 -β t+1 1 θ t+1 = θ t -α t m t+1 √ v t+1 + b t+1 NADAM(H t , α t , β 1 , β 2 , ) m 0 = 0, v 0 = 0 m t+1 = β 1 m t + (1 -β 1 )∇ (θ t ) v t+1 = β 2 v t + (1 -β 2 )∇ (θ t ) 2 b t+1 = 1 -β t+1 2 1 -β t+1 1 θ t+1 = θ t -α t β 1 m t+1 + (1 -β 1 )∇ (θ t ) √ v t+1 + b t+1

Definition 1 (Inclusion relationship). Let M, N be update rules for use in a first-order optimization method. M is a subset or specialization of N , if for all φ : N → R n , there exists a sequence ψ i : N → R m , such that for all t ∈ [0, ∞) and histories H t ,

lim i→∞ N (H t , ψ i t ) = M(H t , φ t )

This is denoted M ⊆ N , with equality M = N iff M ⊆ N and N ⊆ M.

Evidently SGD ⊆ MOMENTUM, since SGD(H t , η t ) = MOMENTUM(H t , η t , 0). Many wellknown optimizers fall naturally into this taxonomy. In particular, we consider RMSPROP with momentum (Tieleman and Hinton, 2012) , ADAM (Kingma and Ba, 2015) and NADAM (Dozat, 2016) (see Table 1 ) and show the following inclusions in the appendix.

SGD ⊆ MOMENTUM ⊆ RMSPROP SGD ⊆ MOMENTUM ⊆ ADAM SGD ⊆ NESTEROV ⊆ NADAM (3)

Note, some of these inclusions make use of the flexibility of hyperparameter schedules (dependence of ψ i on t). In particular, to approximate MOMENTUM with ADAM, one needs to choose a learning rate schedule that accounts for ADAM's bias correction.

If two optimizers have an inclusion relationship, the more general optimizer can never be worse with respect to any metric of interest, provided the hyperparameters are sufficiently tuned to optimize that metric. Optimally-tuned MOMENTUM cannot underperform optimally-tuned SGD, because setting γ = 0 in MOMENTUM recovers SGD. However, optimizers with more hyperparameters might be more expensive to tune, so we should have a theoretical or experimental reason for using (or creating) a more general optimizer. For example, MOMENTUM improves local convergence rates over SGD on twice-differentiable functions that are smooth and strongly convex (Polyak, 1964) , and NESTEROV has globally optimal convergence rates within the class of smooth and strongly convex functions (Nesterov, 1983; 2018) .

At first glance, the taxonomy of optimizer inclusions appears to resolve many optimizer comparison questions. However, for a deep learning practitioner, there is no guarantee that the inclusion hierarchy is at all meaningful in practice. For example, the hyperparameters that allow ADAM to match or outperform MOMENTUM might not be easily accessible. They might exist only in the limit of very large values, or be so difficult to find that only practitioners with huge computational budgets can hope to discover them. Indeed, empirical studies and conventional wisdom hold that the inclusion hierarchy does not predict optimizer performance for many practical workloads (Wilson et al., 2017; Balles and Hennig, 2017; Schneider et al., 2019) . Either these experimental investigations are too limited or the taxonomy of this section is of limited practical interest and provides no guidance about which optimizer to use on a real workload. In the following section we attempt to answer this question experimentally, and show that these inclusion relationships are meaningful in practice.

4. EXPERIMENTS:

An empirical comparison of optimizers should aim to inform a careful practitioner. Accordingly, we model our protocol on a practitioner that is allowed to vary all optimization hyperparameters for each optimizer (e.g. α t , β 1 , β 2 , for ADAM) in addition to a parameterized learning rate decay schedule, in contrast to studies that fix a subset of the optimization hyperparameters to their default values (e.g. Wilson et al., 2017; Schneider et al., 2019) . There is no standard method for selecting the values of these hyperparameters, but most practitioners tune at least a subset of the optimization hyperparameters by running a set of trials to maximize performance over the validation set. In our experiments, we run tens to hundreds of individual trials per workload. Given the variety of workloads we consider, this trial budget covers a wide range of computational budgets.

Selecting the hyperparameter search space for each optimizer is a key methodological choice for any empirical comparison of optimizers. Prior studies have attempted to treat each optimizer fairly by using the same search space for all optimizers (e.g. Wilson et al., 2017; Schneider et al., 2019) . However, this requires the assumption that similarly-named hyperparameters should take similar values between optimizers, which is not always true. For example, MOMENTUM and NESTEROV both have similar-looking momentum and learning rate hyperparameters, but NESTEROV tolerates larger values of its momentum hyperparameter (Sutskever et al., 2013) , so any fixed search space will likely be more favorable for one of the two. The situation worsens with less closely related optimizers, and designing a search space that is equally appropriate for optimizers with incommensurate hyperparameters is almost impossible. Despite coming with its own set of challenges, it is most informative to compare optimizers assuming the practitioner is allowed to tune hyperparameters for different optimizers independently by way of optimizer-specific search spaces.

In our experiments, we chose the search space for each optimizer by running an initial set of experiments over a relatively large search space. In a typical case, we ran a single set of initial trials per optimizer to select the final search space. However, in some cases we chose the initial search space poorly, so we ran another set of experiments to select the final search space. The effort required to choose each search space cannot simply be quantified by the number of initial trials; the provenance of each search space is difficult to trace exactly. In some cases, our search spaces were informed by published results or prior experience with particular models and optimizers. We note that this is true of all search spaces in the literature: they are hard-won treasures that tend to be refined over many experiments and across many workloads, representing the sum total of our community's experience. We validated our search spaces by checking that that the optimal hyperparameter values were away from the search space boundaries for all optimizers in all experiments (see Figure 5 in Appendix E). We provide our final search spaces for all experiments in Appendix D. The fact that our final error rates compare favorably to prior published results -including reaching state-of-the-art for our particular configuration of ResNet-50 on ImageNet (see Section 4.2) -supports our claim that our methodology is highly competitive with expert tuning procedures. (Russakovsky et al., 2015) . For language modeling, we trained a 2-layer LSTM model (Hochreiter and Schmidhuber, 1997 ) on Tolstoy's War and Peace; and Transformer (Vaswani et al., 2017) on LM1B (Chelba et al., 2014) . We used a linear learning rate decay schedule parameterized the same way as Shallue et al. ( 2019) for all workloads. We used a fixed batch size and a fixed budget of training steps for each workload independent of the optimizer. Table 2 summarizes these workloads and Appendix B provides the full details.

Given a hypercube-shaped search space, our tuning protocol sought to model a practitioner with a fixed budget of trials trying to achieve the best outcome using tens of feasible trials (10, 50, or 100 depending on the workload).2 A feasible trial is any trial that achieves finite training loss. We used quasi-random uniform search (Bousquet et al., 2017) , and continued the search until we obtained a fixed number of feasible trials. From those trials we considered two statistics. The first, in order to characterize the best outcome, is a metric of interest (e.g. test accuracy) corresponding to the trial achieving the optimum of some other metric (e.g. validation accuracy). The second, in order to characterize the speed of training, is the number of steps required to reach a fixed validation target conditional on at least one trial in the search having reached that target. We chose the target for each workload based on initial experiments and known values from the literature (see Table 2 ). We estimated means and uncertainties using the bootstrap procedure described in Appendix C.

4.2. INCLUSION RELATIONSHIPS MATTER IN PRACTICE:

Figure 1 shows the final predictive performance of six optimizers on four different workloads after tuning hyperparameters to minimize validation error. Regardless of whether we compare final validation error or test error, the inclusion relationships hold in all cases -a more general optimizer never underperforms any of its specializations within the error bars. Similar results hold for training error (see Figure 9 in Appendix E). Training speed is also an important consideration, and Figure 2 demonstrates that the inclusion relationships also hold within error bars when we compare the number of steps required to reach a target validation error. Moreover, these results confirming the relevance of optimizer inclusion relationships do not depend on the exact step budgets or error targets we chose (see Figure 10 in Appendix E), although large changes to these values would require new experiments. 2 for the error targets).

Of course, just because a more general optimizer is no worse than any of its specializations doesn't mean the choice of optimizer makes a large difference on all workloads. For some workloads in Figures 1 and 2, all optimizers perform about the same, while other workloads have a clear ranking or even dramatic differences. For example, the choice of optimizer seems to make little difference for ResNet-32 on CIFAR-10; all optimizers achieve similar predictive performance and training speed.

On the other hand, Transformer on LM1B exhibits a clear ranking in terms of predictive performance and training speed. For this workload, ADAM needs roughly half the steps that MOMENTUM requires to reach our target error, and, although not shown in Figure 2 , roughly six times fewer steps to get the same result as SGD. These differences are clearly significant enough to matter to a practitioner, and highlight the practical importance of choosing the right optimizer for some workloads.

The most general optimizers we considered were RMSPROP, ADAM, and NADAM, which do not include each other as special cases, and whose relative performance is not predicted by inclusion relationships. Across the workloads we considered, none of these optimizers emerged as the clear winner, although ADAM and NADAM generally seemed to have an edge over RMSPROP. For all of these optimizers, we sometimes had to set the parameter orders of magnitude larger than the default value in order to get good results. In particular, we achieved a validation accuracy of 77.1% for ResNet-50 on ImageNet using NADAM with = 9475, a result that exceeds the 76.5% achieved by Goyal et al. (2017) using MOMENTUM. Across just these 4 workloads, the range of the optimal values of the parameter spanned 10 orders of magnitude. Faced with this reality, a practitioner might reasonably doubt their ability to find a value near the optimum. However, we found that we could reasonably expect to find a suitable value with only tens of trials. When tuning for ADAM or NADAM over a large range, we found it more efficient to search over ( , α 0 / ) instead of ( , α 0 ); see Appendix D for more details. 2017). Tuning a subset of optimizer hyperparameters and the initial learning rate is sufficient to equalize performance between all optimizers (left). More extensive hyperparameter tuning in our setup, including the learning rate schedule, improves results for all optimizers and still does not produce any differences between optimizer performances (right).

4.3. RECONCILING DISAGREEMENTS WITH PREVIOUS WORK:

In order to confirm that differences in hyperparameter tuning protocols explain the differences between our conclusions and those of Wilson et al. ( 2017) and Schneider et al. (2019) , we reproduced a representative subset of their results and then inverted, or at least collapsed, the ranking over optimizers just by expanding the hyperparameter search space.

The left pane of Figure 3 shows our experiments on VGG on CIFAR-10 using code released by Wilson et al. (2017) . When we match their protocol and perform their grid search over the initial learning rate and no other tuning, we reproduce their original result showing worse test error for RMSPROP and ADAM. However, when we tune the momentum parameter and with random search, all four optimizers reach nearly identical test error rates. 3 With our learning rate schedule search space, merely tuning the learning rate schedule was enough to make all optimizers reach the same test error within error bars. When we additionally tuned the optimization hyperparameters and weight decay in our setup we also get similar results for all optimizers, removing any evidence the inclusion relationships might be violated in practice.

Figure 4 shows our results with different tuning protocols for a CNN on CIFAR-100 and an LSTM language model trained on War and Peace to match the experiments in Schneider et al. (2019) 2019), we set out to provide a detailed empirical characterization of the optimizer selection process in deep learning. Our central finding is that inclusion relationships between optimizers are meaningful in practice. When tuning all available hyperparameters under a realistic protocol at scales common in deep learning, we find that more general optimizers never underperform their special cases. In particular, we found that RMSPROP, ADAM, and NADAM never underperformed SGD, NESTEROV, or MOMENTUM under our most exhaustive tuning protocol. We did not find consistent trends when comparing optimizers that could not approximate each other. We also found workloads for which there was not a statistically significant separation in the optimizer ranking.

Our experiments have some important limitations and we should be careful not to overgeneralize from our results. The first major caveat is that we did not measure the effects of varying the batch size. Recent empirical work (Shallue et al., 2019; Zhang et al., 2019) has shown that increasing the batch size can increase the gaps between training times for different optimizers, with the gap from SGD to MOMENTUM (Shallue et al., 2019) and from MOMENTUM to ADAM (Zhang et al., 2019) increasing with the batch size. Nevertheless, we strongly suspect that the inclusion relations would be predictive at any batch size under a tuning protocol similar to the one we used. The second important caveat of our results is that they inevitably depend on the tuning protocol and workloads that we considered. Although we made every attempt to conduct realistic experiments, we should only expect our detailed findings to hold for similar workloads under similar protocols, namely uniform quasi-random tuning for tens to hundreds of trials, over hypercube search spaces, and with our specific learning rate schedule parameterization. Nevertheless, these caveats reinforce our central point: all empirical comparisons of neural network optimizers depend heavily on the hyperparameter tuning protocol, perhaps far more than we are used to with comparisons between model architectures.

If we were to extract "best practices" from our findings, then we suggest the following. If we can afford tens or more runs of our code, we should tune all of the hyperparameters of the popular adaptive gradient methods. Just because two hyperparameters have a similar role in two different update rules doesn't mean they should take similar values-optimization hyperparameters tend to be coupled and the optimal value for one may depend on how the others are set. Our results also confirm that the optimal value of Adam's is problem-dependent, so the onus is on empirical studies that fix = 10 -8 to defend that choice. Finally, we should be skeptical of empirical comparisons of optimizers in papers, especially if an optimizer underperforms any of its specializations. When we do inevitably compare optimizers, we should report search spaces and highlight decisions about what hyperparameters were tuned when interpreting results.

A OPTIMIZER INCLUSIONS:

Table 1 summarizes the update rules for the optimizers we consider in this work. We assume update rules as implemented in TensorFlow r1.15. RMSPROP includes momentum. Here we prove the their inclusion relationships, see Definition 1.

MOMENTUM can exactly implement SGD MOMENTUM(I t , η t , 0) = SGD(I t , η t ), so SGD ⊆ MOMENTUM.

NESTEROV can exactly implement SGD NESTEROV(I t , η t , 0) = SGD(I t , η t ), so SGD ⊆ NESTEROV.

RMSPROP with momentum can exactly implement MOMENTUM:

Consider RMSPROP(I t , η t , γ, ρ = 1, = 0), so that

m t+1 = γm t + η t ∇ (θ t ) , θ t+1 = θ t -m t+1 .

This is equivalent to MOMENTUM, since

m (RMSPROP) t+1 ≡ η t v (MOMENTUM) t+1 .

Thus RMSPROP(I t , η t , γ, 1, 0) = MOMENTUM(I t , η t , γ), so MOMENTUM ⊆ RMSPROP.

RMSTEROV can exactly implement NESTEROV:

Consider RMSTEROV(I t , η t , γ, ρ = 1, = 0), so that

m t+1 = γm t + η t ∇ (θ t ) , θ t+1 = θ t -[γm t+1 + η t ∇ (θ t )] .

This is equivalent to MOMENTUM, since

m (RMSPROP) t+1 ≡ η t v (NESTEROV) t+1 .

Thus RMSTEROV(I t , η t , γ, 1, 0) = MOMENTUM(I t , η t , γ), so MOMENTUM ⊆ RMSTEROV.

ADAM can approximate MOMENTUM for large:

Consider ADAM(I t , α t = η t (1 -γ t ), β 1 = γ, β 2 = 0, ), so that

m t+1 = γm t + (1 -γ)∇ (θ t ) , θ t+1 = θ t - η t (1 -γ) m t+1 |∇ (θ t )|/ + 1 . If is large, so that |∇ (θ t )|/ 1, then m t+1 = γm t + (1 -γ)∇ (θ t ) , θ t+1 = θ t -η t m t+1 1 -γ .

This is equivalent to MOMENTUM, since

m (ADAM) t ≡ (1 -γ) v (MOMENTUM) t .

Thus lim →∞ ADAM(I t , η t (1 -γ t ), γ, 0, ) = MOMENTUM(I t , η t , γ), so MOMENTUM ⊆ ADAM

NADAM can approximate NESTEROV for large:

Consider NADAM(I t , α t = η t (1 -γ t ), β 1 = γ, β 2 = 0, ), so that

m t+1 = γm t + (1 -γ)∇ (θ t ) , θ t+1 = θ t - η t (1 -γ) γm t+1 + (1 -γ)∇ (θ t ) |∇ (θ t )|/ + 1 . If is large, so that |∇ (θ t )|/ 1, then m t+1 = γm t + (1 -γ)∇ (θ t ) , θ t+1 = θ t -η t γm t+1 1 -γ + ∇ (θ t ) .

This is equivalent to NESTEROV, since

m (NADAM) t ≡ (1 -γ) v (NESTEROV) t

Thus lim →∞ NADAM(I t , η t (1 -γ t ), γ, 0, ) = NESTEROV(I t , η t , γ), so NESTEROV ⊆ NADAM.

B WORKLOAD DETAILS:

This section details the datasets and models summarized in Table 2 .

B.1 DATASET DESCRIPTIONS:

For Fashion MNIST, CIFAR-10, ImageNet, and LM1B, our setup was identical to Shallue et al. (2019) except for the image pre-processing details described below. For War and Peace, our setup was identical to the "Tolstoi" dataset of Schneider et al. (2019) .

CIFAR-10/100: We pre-processed images by subtracting the average value across all pixels and channels and dividing by the standard deviation. 4 For experiments with the ResNet-32 and CNN models, we followed the standard data augmentation scheme used in He et al. (2016a) : 4 pixels padded on each side with single random crop from padded image or its horizontal reflection. We did not use random cropping for experiments with VGG for consistency with Wilson et al. (2017) . ResNet is described in He et al. (2016a) . We used the improved residual block described in He et al. (2016b) . We used batch normalization (Ioffe and Szegedy, 2015) with exponential moving average (EMA) decay of 0.997 for ResNet-32, and ghost batch normalization (Hoffer et al., 2017) with ghost batch size of 32 and EMA decay of 0.9 for ResNet-50.

VGG is based on "model C" from Simonyan and Zisserman (2014) . It consists of 13 convolutional layers followed by 3 fully connected hidden layers. We followed the modification used by Wilson et al. ( 2017) with batch normalization layers.

LSTM is a two hidden-layer LSTM model (Hochreiter and Schmidhuber, 1997 ) identical to the model used in Schneider et al. (2019) . It uses 128 embedding dimensions and 128 hidden units.

Transformer is the "base" model described in (Vaswani et al., 2017) . We used it as an autoregressive language model by applying the decoder directly to the sequence of word embeddings for each sentence. Unlike the default implementation, we removed dropout regularization and used separate weight matrices for the input embedding layer and the pre-softmax linear transformation, as we observed these choices led to better performing models.

C ESTIMATING TRIAL OUTCOMES VIA BOOTSTRAP:

Our tuning protocol corresponds to running trials with quasi-random hyperparameter values sampled uniformly from the search space until K feasible trials are obtained, with K depending on the workload. We then select the best trial, based on our statistic of interest, over those K trials.

We used the following bootstrap procedure to estimate means and uncertainties of our tuning protocol. We ran N > K trials, with N depending on the workload. Then, for each bootstrap sample, we resampled the dataset of N trials with replacement and computed our statistic on the first K trials of the resampled dataset. We collected 100 such bootstrap samples each time, and from those computed the means, 5 th percentiles, and 95 th percentiles of the bootstrap distribution. We used this procedure to generate the means and error bars for each plot.

Simple CNN on Fashion MNIST used (K, N ) = (100, 500); ResNet-32 on CIFAR-100 used (K, N ) = (100, 500); ResNet-50 on ImageNet used (K, N ) = (50, 250); Transformer on LM1B used (K, N ) = (50, 100); VGG on CIFAR-10 with our code used (K, N ) = (50, 250) for tuning the learning rate schedule and (K, N ) = (100, 500) for tuning the learning rate schedule, {γ, β 1 , β 2 , }, and L 2 regularization; CNN on CIFAR-10 used (K, N ) = (100, 500); LSTM on War and Peace used (K, N ) = (10, 50) for tuning just the learning rate and (K, N ) = (100, 500) for tuning the learning rate schedule and {γ, β 1 , β 2 , }.

The sole exceptions to this bootstrap procedure are the two left panels of Figure 3 , for which we used a similar procedure to Wilson et al. ( 2017) to ensure comparability. For each optimizer, we selected the trial that minimized validation error in our final search space and ran the same hyperparameter values 5 times, reporting the mean, minimum, and maximum test error over those 5 runs in Figure 3 . This is slightly different to Wilson et al. (2017) , who chose the trial that minimized training error and reported validation error. When tuning the learning rate and {γ, }, we used 24 trials per optimizer in the initial search space (which we used to select the final search space), and 16 trials per optimizer in the final search space.

D HYPERPARAMETER SEARCH SPACES:

When tuning hyperparameters over a large range, we found that our search could sometimes be made more efficient if we parametrized the search space in a way that decorrelated the axes of the space. For example, with MOMENTUM and NESTEROV we observed a clear relationship between the initial learning rate η 0 and the momentum parameter γ; smaller values of η 0 require larger values of γ for good performance, and vice versa. Indeed, Shallue et al. (2019) suggested that these optimizers are governed by the "effective learning rate" η eff = η 0 /(1 -γ), and inspired by this, we found that searching over (η 0 , η eff ) instead of (η 0 , γ) usually led to a more efficient hyperparameter search.

Similarly, with ADAM and NADAM we observed a relationship between the initial learning rate α 0 and the parameter; larger values of α 0 require larger values of for good performance, and vice versa. This is not surprising given the analysis in Appendix A that showed that, for large , α 0 / is analogous to the effective learning rate of ADAM and NADAM. We found that searching over ( , α 0 / ) was usually more efficient than searching over ( , α 0 ). We used these techniques in a subset of our experiments.

Below we report the search spaces used for our experiments. We include both the initial search spaces used to refine the search spaces, and the final spaces used to generate the plots. When only one search space was used, we denote the initial space as final. η 0 , α 0 , 1 -γ, 1 -β 1 , 1 -β 2 , , and combinations thereof are always tuned on a log scale. The number of samples from each search space is specified in Appendix C.

D.1 CNN ON FASHION MNIST:

We used linear learning rate decay for all experiments. We tuned the number of decay steps within [0.5, 1.0] times the number of training steps and the learning rate decay factor within {10 -3 , 10 -2 , 10 -1 }. We did not use L 2 regularization or weight decay.

η 0 initial [10 -2 , 10 2 ] final [10 -2 , 10 1 ] Table 3: SGD η 0 1 -γ initial [10 -4 , 10 2 ] [10 -4 , 1] final [10 -5 , 10 1 ] [10 -4 , 1] Table 4: MOMENTUM η 0 1 -γ initial [10 -4 , 10 2 ] [10 -4 , 1] final [10 -4 , 10 1 ] [10 -4 , 1]

Table 5 : NESTEROV

η 0 1 -γ 1 -ρ initial [10 -4 , 10 1 ] [10 -2 , 1] [10 -4 , 1] [10 -5 , 10 1 ] final [10 -5 , 1] [10 -2 , 1] [10 -3 , 1] [10 -10 , 10 -5 ]

Table 6 : RMSPROP We used linear learning rate decay for all experiments. We tuned the number of decay steps within [0.5, 1.0] times the number of training steps and the learning rate decay factor f within the values shown in the tables below. λ L2 denotes the L 2 regularization coefficient.

α 0 1 -β 1 1 -β 2 initial [10 -4 , 10 -1 ] [10 -3 , 5 × 10 -1 ] [10 -4 , 10 -1 ] [10 -9 , 10 -5 ] final [10 -5 , 10 -1 ] [10 -3 , 1] [10 -4 , 1] [10 -10 , 10 -5 ] Table 7: ADAM α 0 / 1 -β 1 1 -β 2 initial [10 -2 , 10 4 ] [10 -3 , 1] [10 -4 , 1] [10 -10 , 10 10 ] final [10 -1 , 10 1 ] [10 -3 , 1] [10 -4 , 1] [10 -6 , 10 -2 ]

η 0 λ L2 f

final [10 -2 , 10 2 ] {10 -5 , 10 -4 , 10 -3 , 10 -2 } {10 -4 , 10 -3 , 10 -2 , 10 -1 } Table 9 : SGD

η 0 1 -γ λ L2 f final [10 -4 , 10 2 ] [10 -3 , 1] {10 -5 , 10 -4 , 10 -3 , 10 -2 } {10 -4 , 10 -3 , 10 -2 , 10 -1 } Table 10: MOMENTUM η 0 1 -γ λ L2 f initial [10 -4 , 10 2 ] [10 -4 , 10 1 ] 10 -4 {10 , 10 -2 , 10 -1 } final [10 -4 , 10 1 ] [10 -4 , 1] {10 -5 , 10 -4 , 10 -3 , 10 -2 } {10 -4 , 10 -3 , 10 -2 , 10 -1 } Table 11: NESTEROV η 0 1 -γ 1 -ρ λ L2 f initial [10 -4 , 10 1 ] [10 -2 , 1] [10 -4 , 1] [10 -5 , 10 1 ] 10 -4 {10 -3 , 10 -2 , 10 -1 } final [10 -4 , 10 1 ] [10 -3 , 1] [10 -4

, 1] [10 -5 , 10 1 ] {10 -5 , 10 -4 10 -3 , 10 -2 } {10 -4 , 10 -3 10 -2 , 10 -1 } Table 12 : RMSPROP

α 0 1 -β 1 1 -β 2 λ L2 f initial [10 -4 , 10 -1 ] [10 -3 , 5 × 10 -1 ] [10 -4 , 10 -1 ] [10 -9 , 10 -5 ] 10 -4 {10 -3 , 10 -2 10 -1 } final [10 -3 , 10 1 ] [10 -3 , 1] [10 -4 , 10 -1 ] [10 -5 , 10 1 ] {10 -5 , 10 -4 10 -3 , 10 -2 } {10 -4 , 10 -3 10 -2 , 10 -1 } Table 13: ADAM α 0 / 1 -β 1 1 -β 2 λ L2 f initial [10 -2 , 10 4 ] [10 -3 , 1] [10 -4 , 1] [10 -10 , 10 10 ] {10 -5 , 10 -4 10 -3 , 10 -2 } {10 -4 , 10 -3 10 -2 , 10 -1 } final [10 -2 , 1] [10 -3 , 1] [10 -4 , 1] [1, 10 4 ] {10 -5 , 10 -4 10 -3 , 10 -2 } {10 -4 , 10 -3 10 -2 , 10 -1 } Table 14: NADAM D.3 RESNET-50 ON IMAGENET

We used linear learning rate decay for all experiments. We tuned the number of decay steps within [0.5, 1.0] times the number of training steps and the learning rate decay factor f within the values shown in the tables below. λ wd denotes the weight decay coefficient and τ denotes the label smoothing coefficient. We used linear learning rate decay for all experiments. We tuned the number of decay steps within [0.5, 1.0] times the number of training steps and the learning rate decay factor within {10 -4 , 10 -3 , 10 -2 , 10 -1 , 1}. 2017), we decayed the initial learning rate by a factor of 0.5 every 25 epochs and used a fixed L 2 regularization coefficient of 0.0005. We used linear learning rate decay, and tuned the number of decay steps within [0.5, 1.0] times the number of training steps and the learning rate decay factor f within the values shown in the tables below. We consider this search space to be appropriate because the optimal values are away from the search space boundaries.

η 0 λ wd τ f initial [10 -2 , 10 1 ] [10 -5 , 10 -2 ] {0, 10 -2 , 10 -1 } {10 -4 , 10 -3 , 10 -2 , 10 -1 } final [1, 10 2 ] [10 -4 , 10 -3 ] 10 -1 {10 -4 , 10 -3 , 10 -2 , 10 -1 } Table 15: SGD η 0 1 -γ λ wd τ f initial [10 -3 , 1] [10 -3 , 1] [10 -5 , 10 -2 ] {0, 10 -2 , 10 -1 } {10 -4 , 10 -3 , 10 -2 , 10 -1 } final [10 -2 , 1] [10 -2 , 1] [10 -4 , 10 -3 ] 10 -2 {10 -4 , 10 -3 , 10 -2 , 10 -1 } Table 16: MOMENTUM η 0 1 -γ λ wd τ f initial [10 -3 , 1] [10 -3 , 1] [10 -5 , 10 -2 ] {0, 10 -2 , 10 -1 } 10 -3 final [10 -2 , 1] [10 -3 , 1] [10 -4 , 10 -3 ] 0 {10 -4 , 10 -3 , 10 -2 , 10 -1 } Table 17: NESTEROV η 0 / √ 1 -γ 1 -ρ λ wd τ f initial [10 -2 , 10 4 ] 0.1 [10 -4 , 1] [10 -10 , 10 10 ] [10 -5 , 10 -2 ] {0, 10 -2 , 10 -1 } 10 -3 final [10 -2 , 1] 0.1 [10 -2 , 1] [10 -8 , 10 -3 ] [10 -4 , 10 -3 ] 0 {10 -4 , 10 -3 10 -2 , 10 -1 } Table 18: RMSPROP α 0 / 1 -β 1 λ wd τ f initial [1, 10 2 ] [10 -3 , 1] [1, 10 4 ] [10 -5 , 10 -3 ] {0, 10 -2 , 10 -1 } 10 -3 final [1, 10 2 ] [10 -2 , 1] [10 -2 , 10 2 ] 10 -4 10 -1 {10 -4 , 10 -3 10 -2 , 10 -1 } Table 19: ADAM α 0 / 1 -β 1 λ wd τ f initial [10 -1 , 10 3 ] [10 -3 , 1] [10 -2 , 10 10 ] [10 -5 , 10 -2 ] {0, 10 -2 , 10 -1 } 10 -3 final [1, 10 2 ] [10 -3 , 1] [10 3 , 10 7 ] 10 -4 10 -1 10 -3

η 0 final [10 -4 , 10 -1 ] Table 21: SGD η 0 1 -γ final [10 -4 , 10 -1 ] [10 -4 , 1] Table 22: MOMENTUM η 0 1 -γ final [10 -4 , 10 -1 ] [10 -4 , 1] Table 23: NESTEROV η 0 1 -γ 1 -ρ initial [10 -4 , 10 1 ] [10 -2 , 1] [10 -2 , 1] [10 -12 , 10 10 ] final [10 -6 , 10 -2 ] [10 -2 , 1] [10 -3 , 1] [10 -7 , 10 -1 ] Table 24: RMSPROP α 0 1 -β 1 1 -β 2 initial [10 -5 , 10 -2 ] [10 -3 , 5 × 10 -1 ] [10 -4 , 10 -1 ] [10 -9 , 10 -5 ] final [10 -4 , 10 -2 ] [10 -3 , 1] [10 -5 , 10 -1 ] [10 -7 , 10 -2 ] Table 25: ADAM α 0 1 -β 1 1 -β 2 final [10 -5 , 10 -2 ] [10 -3 , 1] [10 -5 , 10 -1 ] [10 -9 , 10 -5 ]

η 0 initial [10 -3 , 1] final [10 -1 , 10 1 ] Table 27: SGD η 0 1 -γ initial [10 -3 , 1] [10 -3 , 1] final [10 -1 , 10 1 ] [10 -1 , 1]

α 0 / 1 -β 1 1 -β 2 initial [10 -1 , 10 1 ] [10 -3 , 1] [10 -4 , 10 -1 ] [10 2 , 10 6 ] final [10 -1 , 10 1 ] [10 -3 , 1] [10 -5 , 10 -2 ] [10 2 , 10 6 ]

η 0 f initial [10 -3 , 10 1 ] {10 -4 , 10 -3 , 10 -2 , 10 -1 } final [1, 10 1 ] {10 -4 , 10 -3 , 10 -2 , 10 -1 } Table 45: SGD η 0 1 -γ f initial [10 -4 , 1] [10 -3 , 1] {10 -4 , 10 -3 , 10 -2 , 10 -1 } final [10 -1 , 10 1 ] [10 -2 , 1] {10 -4 , 10 -3 , 10 -2 , 10 -1 } Table 46: MOMENTUM α 0 / 1 -β 1 1 -β 2 f initial [10 -2 , 10 4 ] [10 -3 , 1] [10 -4 , 10 -1 ] [10 -10 , 10 10 ] {10 -4 , 10 -3 , 10 -2 , 10 -1 } final [1, 10 2 ] [10 -2 , 1] 0.999 [1, 10 4 ] 10 -3

2 0 2 1 2 2 2 3 2 4 2 5 2 6 2 7 3.3 ResNet-32 on CIFAR-10 3.5 4.0 4.5 5.0 5.5 6.0 6.5 7.0 7.5

Step Budget 

:

TensorFlow currently refers to as "a small constant for numerical stability"; https://www. tensorflow.org/versions/r1.15/api_docs/python/tf/train/AdamOptimizer.

Although we used a budget of tens of independent tuning trials throughout this section, in retrospect the best validation error across tuning trials converged quite quickly for our final search spaces, producing good results with fewer than 20 trials in many cases. See Figures6-8in Appendix E.

Wilson et al. (2017) selected trials to minimize the training loss and then report test set results. As Figure3shows, removing this somewhat non-standard choice and tuning on a validation set and reporting test set results does not change anything.

We used the TensorFlow op tf.image.per image standardization.

See https://gist.github.com/ksimonyan/211839e770f7b538e2d8#description for the mean RGB values used.

Figures and Tables:

Figure 2: The relative training speed of optimizers is consistent with the inclusion relationships. We measured (idealized) training speed as the number of training steps required to reach a target validation error (see Table2for the error targets).

Figure3: Tuning more hyperparameters removes the differences in test error between optimizers observed byWilson et al. (2017). Tuning a subset of optimizer hyperparameters and the initial learning rate is sufficient to equalize performance between all optimizers (left). More extensive hyperparameter tuning in our setup, including the learning rate schedule, improves results for all optimizers and still does not produce any differences between optimizer performances (right).

Figure4: Tuning more hyperparameters changes optimizer rankings fromSchneider et al. (2019) to rankings that are consistent with the inclusion relationships. The leftmost columns for each workload reproduce the rankings fromSchneider et al. (2019), while the remaining columns tune over increasingly general search spaces. All columns use our random search tuning protocol.

We augmented images at training time by resizing each image, taking a random crop of 224 × 224 pixels, randomly horizontally reflecting the cropped images, and randomly distorting the image colors. At evaluation time, we performed a single central crop of 224 × 224 pixels. In both training and evaluation, we then subtracted the global mean RGB value from each pixel using the values computed by Simonyan and Zisserman (2014). 5B.2 MODEL DESCRIPTIONSSimple CNN is identical to the base model described inShallue et al. (2019). It consists of 2 convolutional layers with max pooling followed by 1 fully connected layer. The convolutional layers use 5 × 5 filters with stride 1, "same" padding, and ReLU activation function. Max pooling uses a 2 × 2 window with stride 2. Convolutional layers have 32 and 64 filters each and the fully connected layer has 1024 units. It does not use batch normalization.CNN is the "All-CNN-C" model from Springenberg et al. (2014), as used inSchneider et al. (2019). The model consists of 3 convolutional layer blocks with max pooling. The convolutional layers use 5 × 5 filters with stride 1, "same" padding, and ReLU activation function. Max pooling uses a 2 × 2 window with stride 2. Convolutional layer blocks have96, 192 and 192 filters each. As in Schneider  et al. (2019), we used L 2 regularization of 5 × 10 -4 .

Figure 5: Example plot of final validation error projected onto the axes of the hyperparameter space.We consider this search space to be appropriate because the optimal values are away from the search space boundaries.

Figure 6: Validation performance of the best trial mostly converges with as few as 2 4 hyperparameter tuning trials for Transformer on LM1B. Shaded regions indicate 5 th and 95 th percentiles estimated with bootstrap sampling (see Appendix C). The search spaces can be found in Appendix D.4.

Figure 7: Validation performance of the best trial mostly converges with as few as 2 4 hyperparameter tuning trials for ResNet-50 in ImageNet. Shaded regions indicate 5 th and 95 th percentile estimated with bootstrap sampling (see Appendix C). The search spaces can be found in D.3.

Figure 8: Test performance of the best trial mostly converges with as few as 2 3 hyperparameter tuning trials for a 2-layer LSTM on War and Peace. Shaded regions indicate 5 th and 95 th percentile estimated with bootstrap sampling (see Appendix C). The search spaces can be found in D.8.2.

Figure 10: Our results confirming the relevance of optimizer inclusion relationships do not depend on the exact step budgets or error targets we chose.

Update rules considered in this work. SGD is due to

Table content: <table/>

Summary of workloads used in experiments.

Table content: <table><tr><td>Task</td><td>Evaluation metric</td><td>Model</td><td>Dataset</td><td>Target error</td><td>Batch size</td><td>Budget</td></tr><tr><td/><td/><td colspan="3">Simple CNN Fashion MNIST 6.6%</td><td>256</td><td>10k steps</td></tr><tr><td>Image</td><td>Classification</td><td>ResNet-32</td><td>CIFAR-10</td><td>7%</td><td>256</td><td>50k steps</td></tr><tr><td>classification</td><td>error</td><td>CNN</td><td>CIFAR-100</td><td>-</td><td>256</td><td>350 epochs</td></tr><tr><td/><td/><td>VGG-16</td><td>CIFAR-10</td><td>-</td><td>128</td><td>250 epochs</td></tr><tr><td/><td/><td>ResNet-50</td><td>ImageNet</td><td>24%</td><td>1024</td><td>150k steps</td></tr><tr><td>Language modeling</td><td>Classification error</td><td>LSTM</td><td>War and Peace</td><td>-</td><td>50</td><td>200 epochs</td></tr><tr><td/><td colspan="3">Cross entropy Transformer LM1B</td><td>3.45</td><td>256</td><td>750k steps</td></tr><tr><td colspan="2">4.1 OVERVIEW</td><td/><td/><td/><td/><td/></tr></table>

The relative performance of optimizers is consistent with the inclusion relationships, regardless of whether we compare final validation error (top) or test error (bottom). For all workloads, we tuned the hyperparameters of each optimizer separately, and selected the trial that achieved the lowest final validation error.

Table content: <table><tr><td>Validation Error</td><td colspan="2">0.055 0.060 0.065 0.070 0.075 Simple CNN on Fashion MNIST Validation Error</td><td>0.055 0.060 0.065 0.070 0.075 0.080</td><td colspan="2">ResNet-32 on CIFAR-10</td><td>Validation Error</td><td>0.225 0.230 0.235 0.240 0.245</td><td colspan="2">ResNet-50 on ImageNet</td><td>Validation Cross Entropy</td><td>3.3 3.4 3.5 3.6 3.7</td><td>Transformer on LM1B</td></tr><tr><td/><td>0.090</td><td/><td>0.100</td><td/><td/><td/><td>0.220</td><td/><td/><td>3.7</td></tr><tr><td>Test Error</td><td>0.060 0.065 0.070 0.075 0.080 0.085</td><td>Test Error</td><td>0.060 0.070 0.080 0.090</td><td/><td/><td>Test Error</td><td>0.200 0.205 0.210 0.215</td><td/><td/><td>Test Cross Entropy</td><td>3.4 3.5 3.6</td></tr><tr><td/><td>0.055</td><td/><td>0.050</td><td/><td/><td/><td>0.195</td><td/><td/><td>3.3</td></tr><tr><td/><td>SGD</td><td/><td/><td>Momentum</td><td>Nesterov</td><td/><td/><td>RMSProp</td><td>Adam</td><td>NAdam</td></tr><tr><td colspan="2">Figure 1:</td><td/><td/><td/><td/><td/><td/><td/><td/></tr></table>

. As reported bySchneider et al. (2019), if we only tune the learning rate without tuning the decay schedule or other optimizer hyperparameters, ADAM does worse than MOMENTUM for the CNN and SGD performs slightly better than ADAM and MOMENTUM on the War and Peace dataset, althoughSchneider et al. (2019) found a larger advantage for SGD. However, once we tune the all the optimizer hyperparameters, ADAM does better than MOMENTUM which does better than SGD, as predicted by the inclusion relationships.

Table content: <table><tr><td/><td>0.55</td><td/><td colspan="2">CNN on CIFAR-100</td><td/><td/><td/><td>0.42 LSTM on War and Peace</td></tr><tr><td>Final Test Error</td><td>0.35 0.40 0.45 0.50</td><td/><td/><td/><td/><td/><td/><td>0.38 0.39 0.40 0.41</td></tr><tr><td/><td>0.30</td><td>Tuning const. LR</td><td>Tuning LR schedule</td><td colspan="2">Tuning const. LR + {γ, β 1 , β 2 , }</td><td colspan="2">Tuning LR schedule + {γ, β 1 , β 2 , }</td><td>0.37</td><td>Tuning const. LR</td><td>+ {γ, β 1 , β 2 , } LR schedule Tuning</td></tr><tr><td/><td/><td/><td/><td>SGD</td><td colspan="2">Momentum</td><td>Adam</td></tr><tr><td colspan="9">We conclude that the reason both Schneider et al. (2019) and Wilson et al. (2017) observed a ranking</td></tr><tr><td colspan="9">that, at first glance, contradicts the inclusion relationships is because they were not tuning enough</td></tr><tr><td colspan="9">of the hyperparameters. If we recast their results in our terminology where ADAM with default</td></tr><tr><td colspan="9">is a different optimizer than ADAM with tuned then there is no contradiction with our results</td></tr><tr><td colspan="9">and it becomes clear immediately that they do not consider the most interesting form of ADAM for</td></tr><tr><td colspan="3">practitioners.</td><td/><td/><td/><td/><td/></tr></table>



Table content: <table><tr><td>: NADAM</td></tr></table>



Table content: <table><tr><td>: NADAM</td></tr></table>

NADAM D.5 VGG ON CIFAR-10 USING CODE FROM WILSON ET AL. (2017) D.5.1 GRID SEARCH OVER LEARNING RATE We tuned over the same grid of initial learning rate values for each optimizer as Wilson et al. (2017). As in Wilson et al. (2017), we decayed the initial learning rate by a factor of 0.5 every 25 epochs and used a fixed L 2 regularization coefficient of 0.0005.We used our quasi-random tuning protocol to tune over the initial learning rate, MOMENTUM's γ, RMSPROP's , and ADAM's . As inWilson et al. (

Table content: <table><tr><td>D.5.2 TUNING LEARNING RATE &amp; {γ, }</td></tr></table>



Table content: <table><tr><td/><td colspan="2">: MOMENTUM</td></tr><tr><td/><td/><td>α 0 /</td><td>√</td></tr><tr><td colspan="4">initial [10 -10 , 10 10 ] [10 -2 , 10 4 ]</td></tr><tr><td>final</td><td>[10 -2 , 10 2 ]</td><td colspan="2">[10 -1 , 10 1 ]</td></tr><tr><td/><td colspan="2">Table 29: RMSPROP</td></tr><tr><td/><td/><td colspan="2">α 0 /</td></tr><tr><td colspan="4">initial [10 -10 , 10 10 ] [10 -2 , 10 4 ]</td></tr><tr><td>final</td><td>[10 6 , 10 10 ]</td><td colspan="2">[10 -1 , 10 1 ]</td></tr></table>

ADAM

Table content: <table/>



Table content: <table><tr><td>: ADAM</td></tr></table>

ADAM D.8.2 TUNING LEARNING RATE SCHEDULE & {γ, β 1 , β 2 , }

Table content: <table/>



Table content: <table><tr><td>: ADAM</td></tr></table>

Peer review (iclr2020):
The paper provides an empirical comparison of a set of first-order optimization methods for deep learning models. Those optimizers include stochastic gradient descent, momentum method, RMSProp, Adam, Nesterov, and Nadam, which arguably covers all popular variants used in the literature. Although it is not the first empirical study on this topic, its conclusion differs slightly. The conclusion is a rather intuitive one: With proper parameter search, the *richer*, more powerful optimizers tend to work better, regardless of the downstream tasks. Pros: - Intuitive results with a well designed workloads and experiments. For practitioners that want to start their own hyperparameter search, the workloads and setups are likely to be useful. Cons: - I am not entirely convinced that the inclusion relationship is indeed a major cause or indicator of different optimizers* performance. There is no theoretical justification; Empirically, if one takes two optimizers equally rich and tunes one of them more intensively, one should expect a better performance, too. Suggestions: - I think at least the basic definitions of different optimizers should be given in the main text. Otherwise, readers without detailed knowledge of all these optimizers cannot follow the paper. For example, the paper starts talking about the taxonomy of the optimizers with their corresponding hyperparameters in Section 3.2 before giving any functional form of the optimizers. - I would suggest the authors to follow the convention and use the term *hyperparameter* rather than *metaparameter*. The readers of this paper are not primarily Bayesian, there is really no need to divert from the convention. Besides, the term *Bayesian hyperparameter tuning* is widely used even. - I wonder to which extent the network structures impact the choice of the hyperparameter (e.g., CNN vs. RNN).