Descending through a Crowded Valley — Benchmarking Deep Learning Optimizers

1 INTRODUCTION . Large-scale stochastic optimization drives a wide variety of machine learning tasks . Because choosing the right optimization algorithm and effectively tuning its hyperparameters heavily influences the training speed and final performance of the learned model , doing so is an important , every-day challenge to practitioners . Hence , stochastic optimization methods have been a focal point of research ( cf . Figure 1 ) , engendering an ever-growing list of algorithms , many of them specifically targeted towards deep learning . The hypothetical machine learning practitioner who is able to keep up with the literature now has the choice among hundreds of methods ( cf . Table 2 in the appendix ) —each with their own set of tunable hyperparameters—when deciding how to train their model . There is limited theoretical analysis that would clearly favor one of these choices over the others . Some authors have offered empirical comparisons on comparably small sets of popular methods ( e.g . Wilson et al. , 2017 ; Choi et al. , 2019 ; Sivaprasad et al. , 2020 ) ; but for most algorithms , the only formal empirical evaluation is offered by the original work introducing the method . Many practitioners and researchers , meanwhile , rely on personal and anecdotal experience , and informal discussion on social media or with colleagues . The result is an often unclear , perennially changing “ state of the art ” occasionally driven by hype . The key obstacle for an objective benchmark is the combinatorial cost of such an endeavor posed by comparing a large number of methods on a large number of problems , with the high resource and time cost of tuning each method ’ s parameters and repeating each ( stochastic ) experiment repeatedly for fidelity . Offering our best attempt to construct such a comparison , we conduct a large-scale benchmark of optimizers to further the debate about deep learning optimizers , and to help understand how the choice of optimization method and hyperparameters influences the training performance . Specifically , 1https : //github.com/AnonSubmitter3/Submission543 we examine whether recently proposed methods show an improved performance compared to more established methods such as SGD or ADAM . Additionally , we are interested in assessing whether optimization methods with well-working default hyperparameters exist that are able to keep up with tuned optimization methods . To this end , we evaluate more than a dozen optimization algorithms , largely selected for their perceived popularity , on a range of representative deep learning problems ( see Figure 4 ) drawing conclusions from tens of thousands of individual training runs . Right up front , we want to state clearly that it is impossible to include all optimizers ( cf . Table 2 in the appendix ) , and to satisfy any and all expectations readers may have on tuning and initialization procedures , or the choice of benchmark problems—not least because everyone has different expectations in this regard . In our personal opinion , what is needed is an empirical comparison by a third party not involved in the original works . As a model reader of our work , we assume a careful practitioner who does not have access to near-limitless resources , nor to a broad range of personal experiences . As such , the core contributions ( in order of appearance , not importance ) of our work are : A concise summary of optimization algorithms and schedules A partly automated , mostly manual literature review provides a compact but extensive list of recent advances in stochastic optimization . We identify more than a hundred optimization algorithms ( cf . Table 2 in the appendix ) and more than 20 families of hyperparameter schedules ( cf . Table 3 in the appendix ) published at least as pre-prints . An extensive optimizer benchmark on deep learning tasks We conduct a large-scale optimizer benchmark , specifically focusing on optimization problems arising in deep learning . We evaluate 14 optimizers on eight deep learning problems using four different schedules , tuning over dozens of hyperparameter settings , to our knowledge , this is the most comprehensive empirical evaluation of deep learning optimizers to date ( cf . Section 1.1 on related work ) . An analysis of thousands of optimization runs Our empirical experiments indicate that an optimizer ’ s performance highly depends on the test problem ( see Figure 4 ) . But some high-level trends emerge , too : ( 1 ) Evaluating multiple optimizers with default hyperparameters works approximately as well as tuning the hyperparameters for a fixed optimizer . ( 2 ) Using an additional untuned learning rate schedule helps on average , but its effect varies greatly depending on the optimizer and the test problem . ( 3 ) While there is no optimizer that clearly dominates across all tested workloads , some of the algorithms we tested exhibited highly variable performance . Others demonstrated decent performance consistently . We deliberately refrain from recommending a single one among them , because we could not find a clear winner with statistical confidence . An open-source baseline for future optimizer benchmarks Our results are accessible online in an open and easily accessible form ( see footnote on Page 1 ) . These results can thus be used as competitive and well-tuned baselines for future benchmarks of new algorithms , drastically reducing the amount of computational budget required for a meaningful optimizer comparison . Our baselines can easily be expanded , and we encourage others to contribute to this collection . The high-level result of our benchmark is , perhaps expectedly , not a clear winner . Instead , our comparison shows that , while some optimizers are frequently decent , they also generally perform similarly , switching their relative positions in the ranking which can partially be explained by the No Free Lunch Theorem ( Wolpert & Macready , 1997 ) . A key insight of our comparison is that a practitioner with a new deep learning task can expect to do about equally well by taking almost any method from our benchmark and tuning it , as they would by investing the same computational resources into running a set of optimizers with their default settings and picking the winner . Possibly the most important takeaway from our comparison is that “ there are now enough optimizers. ” Methods research in stochastic optimization should focus on significant ( conceptual , functional , performance ) improvements—such as methods specifically suited for certain problem types , innerloop parameter tuning or structurally novel methods . We make this claim not to discourage research but , quite on the contrary , to offer a motivation for more meaningful , non-incremental research . 1.1 RELATED WORK . Following the rapid increase in publications on optimizers , benchmarking these methods for the application in deep learning has only recently attracted significant interest . Schneider et al . ( 2019 ) introduced a benchmarking framework called DEEPOBS , which includes a wide range of realistic deep learning test problems together with standardized procedures for evaluating optimizers . Metz et al . ( 2020 ) presented TASKSET , another collection of optimization problems focusing on smaller but many more test problems . For the empirical analysis presented here , we use DEEPOBS as it provides optimization problems closer to real-world deep learning tasks . In contrast to our evaluation of existing methods , TASKSET and its analysis focuses on meta-learning new algorithms or hyperparameters . Both Choi et al . ( 2019 ) and Sivaprasad et al . ( 2020 ) analyzed specific aspects of benchmarking process . Sivaprasad et al . ( 2020 ) used DEEPOBS to illustrate that the relative performance of an optimizer depends significantly on the used hyperparameter tuning budget . The analysis by Choi et al . ( 2019 ) supports this point , stating that “ the hyperparameter search space may be the single most important factor explaining the rankings. ” They further stress a hierarchy among optimizers , demonstrating that , given sufficient hyperparameter tuning , more general optimizers can never be outperformed by special cases . In their study , however , they manually chose a hyperparameter search space per optimizer and test problem basing it either on prior published results , prior experiences , or pre-tuning trials . Here we instead aim to identify well-performing optimizers in the case of a less extensive tuning budget and especially when there is no prior knowledge about well-working hyperparameter values for each specific test problem . We further elaborate on the influence of our chosen hyperparameter search strategy in Section 4 discussing the limitations of our empirical study . Our work is also related to empirical generalization studies of adaptive methods , such as that of Wilson et al . ( 2017 ) which sparked an extensive discussion whether adaptive methods ( e.g . ADAM ) tend to generalize worse than standard first-order methods ( i.e . SGD ) . 2 BENCHMARKING PROCESS . Any benchmarking effort requires tricky decisions on the experimental setup that influence the result . Evaluating on a specific task or picking a certain tuning budget , for example , may favor or disadvantage certain algorithms ( Sivaprasad et al. , 2020 ) . It is impossible to avoid these decisions or to cover all possible choices . Aiming for generality , we evaluate the performance on eight diverse real-world deep learning problems from different disciplines ( Section 2.1 ) . From a collection of more than a hundred deep learning optimizers ( Table 2 in the appendix ) we select 14 of the most popular and most promising choices ( cf . Figure 1 ) for this benchmark ( Section 2.2 ) . For each test problem and optimizer we evaluate all possible combinations of three different tuning budgets ( Section 2.3 ) and four selected learning rate schedules ( Section 2.4 ) , thus covering the following combinatorial space : Problem P1 P2 . . . P8 8 × Optimizer AMSBound AMSGrad . . . SGD 14 × Tuning one-shot small budget large budget 3 × Schedule constant cosine decay cosine warm restarts trapezoidal 4 . Combining those options results in 1,344 possible configurations and roughly 35,000 individual runs . 2.1 TEST PROBLEMS . We consider the eight optimization tasks summarized in Table 1 , available as the “ small ” ( P1–P4 ) and “ large ” ( P5–P8 ) problem sets , respectively , together forming the default collection of DEEPOBS . A detailed description of these problems , including architectures , training parameters , etc . can be found in the work of Schneider et al . ( 2019 ) .2 DEEPOBS ’ test problems provide several performance metrics , including the training and test loss , the validation accuracy , etc . While these are all relevant , any comparative evaluation of optimizers requires picking only a few , if not just one particular performance metric . For our analysis ( Section 3 ) , we focus on the final test accuracy ( or the final test loss , if no accuracy is defined for this problem ) . This metric captures , for example , the optimizer ’ s ability to generalize and is thus highly relevant for practical use . Our publicly released results include all metrics for completeness . An example of training loss performance is shown in Figure 16 in the appendix . Accordingly , the tuning ( Section 2.3 ) is done with respect to the validation metric . We discuss possible limitations resulting from these choices in Section 4 . 2.2 OPTIMIZER SELECTION . In Table 2 in the appendix we collect over a hundred optimizers introduced for , suggested for , or used in deep learning . This list was manually and incrementally collected by multiple researchers trying to keep up with the field over recent years . It is thus necessarily incomplete , although it may well represent one of the most exhaustive of such collections . Even this incomplete list , though , contains too many entries for a meaningful benchmark with the degrees of freedom collected above . This is a serious problem for research : Even an author of a new optimizer , let alone a practitioner , could not possibly be expected to compare their work with every possible competing method . We thus selected a subset of 14 optimizers , which we consider to be currently the most popular choices in the community ( see Table 4 in the appendix ) . These do not necessarily reflect the “ best ” algorithms , but are either commonly used by practitioners and researchers , or have recently generated enough attention to garner interest . Our selection is focused on first-order optimization methods , both due to their prevalence for non-convex continuous optimization problems in deep learning as well as to simplify the comparison . Whether there is a significant difference between these optimizers or if they are inherently redundant is one of the questions this work investigates . With our list , we tried to focus on optimization algorithms over techniques , although we acknowledge , the line being very blurry . Techniques such as averaging weights ( Izmailov et al. , 2018 , e.g . ) or ensemble methods ( Garipov et al. , 2018 , e.g . ) have been shown to be simple but effective at improving the optimization performance . Those methods , however , can be applied to all methods in our lists , similar to regularization techniques , learning rate schedules , or tuning methods and we have , therefore , decided to omit them from Table 2 . 2All experiments were performed using version 1.2.0-beta of DEEPOBS and TensorFlow version 1.15 Abadi et al . ( 2015 ) .

Summary: This paper benchmarks popular optimizers for training neural networks. The experiments consider all possible combinations of 3 different tuning budgets, and 4 different fixed learning rate schedules on 8 deep learning workloads for 14 optimizers. The paper highlights two main observations: 1) there is no clear dominating optimizer, and 2) selecting from a pool of optimizers with their default parameters is often as good as tuning a fixed optimizer.