DeepOBS: A Deep Learning Optimizer Benchmark Suite

There is significant past and ongoing research on optimization methods for deep learning . Yet , perhaps surprisingly , there is no generally agreed-upon protocol for the quantitative and reproducible evaluation of such optimizers . We suggest routines and benchmarks for stochastic optimization , with special focus on the unique aspects of deep learning , such as stochasticity , tunability and generalization . As the primary contribution , we present DEEPOBS , a Python package of deep learning optimization benchmarks . The package addresses key challenges in the quantitative assessment of stochastic optimizers , and automates most steps of benchmarking . The library includes a wide and extensible set of ready-to-use realistic optimization problems , such as training Residual Networks for image classification on IMAGENET or character-level language prediction models , as well as popular classics like MNIST and CIFAR-10 . The package also provides realistic baseline results for the most popular optimizers on these test problems , ensuring a fair comparison to the competition when benchmarking new optimizers , and without having to run costly experiments . It comes with output back-ends that directly produce LATEX code for inclusion in academic publications . It supports TENSORFLOW and is available open source . 1 INTRODUCTION . As deep learning has become mainstream , research on aspects like architectures ( Graves et al. , 2014 ; He et al. , 2016 ; Szegedy et al. , 2017 ; Vaswani et al. , 2017 ; Sabour et al. , 2017 ) and hardware ( Ovtcharov et al. , 2015 ; Chen et al. , 2016 ; Reagen et al. , 2016 ; Jouppi , 2016 ) has exploded , and helped professionalize the field . In comparison , the optimization routines used to train deep nets have arguable changed only little . Comparably simple first-order methods like SGD ( Robbins & Monro , 1951 ) , its momentum variants ( MOMENTUM ) ( Polyak , 1964 ; Nesterov , 1983 ) and ADAM ( Kingma & Ba , 2015 ) remain standards ( Goodfellow et al. , 2016 ; Karpathy , 2017 ) . The low practical relevance of more advanced optimization methods is not for lack of research , though . There is a host of papers proposing new ideas for acceleration of first-order methods ( Duchi et al. , 2011 ; Tieleman & Hinton , 2012 ; Zeiler , 2012 ; Dozat , 2016 ; Bello et al. , 2017 ; Loshchilov & Hutter , 2017 ; Reddi et al. , 2018 ) , incorporation of second-order information ( Martens , 2010 ; Martens & Grosse , 2015 ; Botev et al. , 2017 ; Zhang et al. , 2017 ; Chen et al. , 2018 ) , and automating optimization ( Schaul et al. , 2013b ; Mahsereci & Hennig , 2017 ; Rolinek & Martius , 2018 ) , to name just a few . One problem is that these methods are algorithmically involved and difficult to reproduce by practitioners . If they are not provided in packages for popular frameworks like TENSORFLOW , PYTORCH etc. , they get little traction . Another problem , which we hope to address here , is that new optimization routines are often not convincingly compared to simpler alternatives in research papers , so practitioners are left wondering which of the many new choices is the best ( and which ones even really work in the first place ) . Designing an empirical protocol for deep learning optimizers is not straightforward , and the corresponding experiments can be time-consuming . This is partly due to the idiosyncrasies of the domain : • Generalization : While the optimization algorithm ( should ) only ever see the training-set , the practitioner cares about performance of the trained model on the test set . Worse , in some important application domains , the optimizer ’ s loss function is not the objective we ultimately care about . For instance in image classification , the real interest may be in the percentage of correctly labeled images , the accuracy . Since this 0-1 loss is infeasible in practice ( Marcotte & Savard , 1992 ) , a surrogate loss function is used instead . So which score should actually be presented in a comparison of optimizers ? Train loss , because that is what the optimizer actually works on ; test loss , because an over-fitting optimizer is useless , or test accuracy , because that ’ s what the human user cares about ? • Stochasticity : Sub-sampling ( batching ) the data-set to compute estimates of the loss function and its gradient introduces stochasticity . Thus , when an optimizer is run only once on a given problem , its performance may be misleading due to random fluctuations . The same stochasticity also causes many optimization algorithms to have one or several tuning parameters ( learning rates , etc. ) . How should an optimizer with two free parameter be compared in a fair way with one that has only one , or even no free parameters ? • Realistic Settings , Fair Competition : There is a widely-held belief that popular standards like MNIST and CIFAR-10 are too simplistic to serve as a realistic place-holder for a contemporary combination of large-scale data set and architecture . While this worry is not unfounded , researchers , ourselves included , have sometimes found it hard to satisfy the demands of reviewers for ever new data sets and architectures . Finding and preparing such data sets and building a reasonable architecture for them is time-consuming for researchers who want to focus on their novel algorithm . Even when this is done , one then has to not just run one ’ s own algorithm , but also various competing baselines , like SGD , MOMENTUM , ADAM , etc . This step does not just cost time , it also poses a risk of bias , as the competition invariably receives less care than one ’ s own method . Reviewers and readers can never be quite sure that an author has not tried a bit too much to make their own method look good , either by choosing a convenient training problem , or by neglecting to tune the competition . To address these problems , we propose an extensible , open-source benchmark specifically for optimization methods on deep learning architectures . We make the following three contributions : • A protocol for benchmarking stochastic optimizers . Section 2 discusses and recommends best practices for the evaluation of deep learning optimizers . We define three key performance indicators : final performance , speed , and tunability , and suggest means of measuring all three in practice . We provide evidence that it is necessary to show the results of multiple runs in order to get a realistic assessment . Finally , we strongly recommend reporting both loss and accuracy , for both training and test set , when demonstrating a new optimizer as there is no obvious way those four learning curves are connected in general . • DEEPOBS1 , a deep learning optimizer benchmark suite . We have distilled the above ideas into an open-source python package , written in TENSORFLOW ( Abadi et al. , 2015 ) , which automates most of the steps presented in section 2 . The package currently provides over twenty off-the-shelf test problems across four application domains , including image classification and natural language processing , and this collection can be extended and adapted as the field makes progress . The test problems range in complexity from stochastic two dimensional functions to contemporary deep neural networks capable of delivering near state-of-the-art results on data sets such as IMAGENET . The package is easy to install in python , using the pip toolchain . It automatically downloads data sets , sets up models , and provides a back-end to automatically produce LATEX code that can directly be included in academic publications . This automation does not just save time , it also helps researchers to create reproducible , comparable , and interpretable results . • Benchmark of popular optimizers From the collection of test problems , two sets , of four simple ( “ small ” ) and four more demanding ( “ large ” ) problems , respectively , are selected as a core set of benchmarks . Researchers can design their algorithm in rapid iterations on the simpler set , then test on the more demanding set . We argue that this protocol saves time , while also reducing the risk of over-fitting in the algorithm design loop . The package also provides realistic baselines results for the most popular optimizers on those test problems . 1Code available at https : //github.com/fsschneider/deepobs . In Section 4 we report on the performance of SGD , SGD with momentum ( MOMENTUM ) and ADAM on the small and large benchmarks ( this also demonstrates the output of the benchmark ) . For each optimizer we perform an exhaustive but realistic hyperparameter search . The best performing results are provided with DEEPOBS and can be used as a fair performance metric for new optimizers without the need to compute these baselines again . We invite the authors of other algorithms to add their own method to the benchmark ( via a git pull-request ) . We hope that the benchmark will offer a common platform , allowing researchers to publicise their algorithms , giving practitioners a clear view on the state of the art , and helping the field to more rapidly make progress . 1.1 RELATED WORKS . To our knowledge , there is currently no commonly accepted benchmark for optimization algorithms that is well adapted to the deep learning setting . This impression is corroborated by a more or less random sample of recent research papers on deep learning optimization ( Duchi et al. , 2011 ; Zeiler , 2012 ; Kingma & Ba , 2015 ; Martens & Grosse , 2015 ; Dozat , 2016 ; Bello et al. , 2017 ; Loshchilov & Hutter , 2017 ; Reddi et al. , 2018 ) , whose empirical sections follow no joint standard ( beyond a popularity of the MNIST data set ) . There are a number of existing benchmarks for deep learning as such . However , they do not focus on the optimizer . Instead , they are either framework or hardwarespecific , or cover deep learning as a holistic process , wrapping together architecture , hardware and training procedure , The following are among most popular ones : DAWNBench The task in this challenge is to train a model for IMAGENET , CIFAR-10 or SQUAD ( Rajpurkar et al. , 2018 ) as quickly as possible to a specified validation accuracy , tuning the entire tool-chain from architecture to hardware and optimizer ( Coleman et al. , 2017 ) . MLPerf is another holistic benchmark similar to DAWNBench . It has two different rule sets ; only the ‘ open ’ set allows a choice of optimization algorithm ( MLPerf , 2018 ) . Deep Learning Frameworks ( Comparison ) compares runtimes of different high-level frameworks ( Microsoft Machine Learning , 2018 ) . DLBS is a benchmark focused on the performance of deep learning models on various hardware systems with various software ( Hewlett Packard Enterprise , 2017 ) . DeepBench tests the speed of hardware for the low-level operations of deep learning , like matrix products and convolutions ( Baidu Research , 2016 ) . Fathom is another hardware-centric benchmark , which among other things assesses how computational resources are spent ( Adolf et al. , 2016 ) . TBD focuses on the performance of three deep learning frameworks ( Zhu et al. , 2018 ) . None of these benchmarks are good test beds for optimization research . Schaul et al . ( 2013a ) defined unit tests for stochastic optimization . In contrast to the present work , they focus on small-scale problems like quadratic bowls and cliffs . In the context of deep learning , these problems provide unit tests , but do not give a realistic impression of an algorithm ’ s performance in practice .

Summary: The authors propose a benchmark for optimization algorithms specific to deep learning called DeepOBS. They provide code to evaluate an optimizer against a suite of standard tasks in deep learning, and provide well tuned baselines for a comparison. The authors discuss important considerations when comparing optimizers, including how to measure speed and tunability of an optimizer, what metric(s) to compare against, and how to deal with stochasticity.