How much progress have we made in neural network training? A New Evaluation Protocol for Benchmarking Optimizers

1 INTRODUCTION . Due to the enormous data size and non-convexity , stochastic optimization algorithms have become widely used in training deep neural networks . In addition to Stochastic Gradient Descent ( SGD ) ( Robbins & Monro , 1951 ) , many variations such as Adagrad ( Duchi et al. , 2011 ) and Adam ( Kingma & Ba , 2014 ) have been proposed . Unlike classical , hyperparameter free optimizers such as gradient descent and Newton ’ s method1 , stochastic optimizers often hold multiple hyperparameters including learning rate and momentum coefficients . Those hyperparameters are critical not only to the speed , but also to the final performance , and are often hard to tune . It is thus non-trivial to benchmark and compare optimizers in deep neural network training . And a benchmarking mechanism that focuses on the peak performance could lead to a false sense of improvement when developing new optimizers without considering tuning efforts . In this paper , we aim to rethink the role of hyperparameter tuning in benchmarking optimizers and develop new benchmarking protocols to reflect their performance in practical tasks better . We then benchmark seven recently proposed and widely used optimizers and study their performance on a wide range of tasks . In the following , we will first briefly review the two existing benchmarking protocols , discuss their pros and cons , and then introduce our contributions . Benchmarking performance under the best hyperparameters A majority of previous benchmarks and comparisons on optimizers are based on the best hyperparameters . Wilson et al . ( 2017 ) ; Shah et al . ( 2018 ) made a comparison of SGD-based methods against adaptive ones under their best hyperparameter configurations . They found that SGD can outperform adaptive methods on several datasets under careful tuning . Most of the benchmarking frameworks for ML training also assume knowing the best hyperparameters for optimizers ( Schneider et al. , 2019 ; Coleman et al. , 2017 ; Zhu et al. , 2018 ) . Also , the popular MLPerf benchmark evaluated the performance of optimizers under the best hyperparameter . It showed that ImageNet and BERT could be trained in 1 minute using the combination of good optimizers , good hyperparameters , and thousands of accelerators . 1The step sizes of gradient descent and Newton ’ s method can be automatically adjusted by a line search procedure ( Nocedal & Wright , 2006 ) . Despite each optimizer ’ s peak performance being evaluated , benchmarking under the best hyperparameters makes the comparison between optimizers unreliable and fails to reflect their practical performance . First , the assumption of knowing the best hyperparameter is unrealistic . In practice , it requires a lot of tuning efforts to find the best hyperparameter , and the tuning efficiency varies greatly for different optimizers . It is also tricky to define the “ best hyperparameter ” , which depends on the hyperparameter searching range and grids . Further , since many of these optimizers are sensitive to hyperparameters , some improvements reported for new optimizers may come from insufficient tuning for previous work . Benchmarking performance with random hyperparameter search It has been pointed out in several papers that tuning hyperparameter needs to be considered in evaluating optimizers ( Schneider et al. , 2019 ; Asi & Duchi , 2019 ) , but having a formal evaluation protocol on this topic is nontrivial . Only recently , two papers Choi et al . ( 2019 ) and Sivaprasad et al . ( 2020 ) take hyperparameter tuning time into account when comparing SGD with Adam/Adagrad . However , their comparisons among optimizers are conducted on random hyperparameter search . We argue that these comparisons could over-emphasize the role of hyperparameter tuning , which could lead to a pessimistic and impractical performance benchmarking for optimizers . This is due to the following reasons : First , in the random search comparison , each bad hyperparameter has to run fully ( e.g. , 200 epochs ) . In practice , a user can always stop the program early for bad hyperparameters if having a limited time budget . For instance , if the learning rate for SGD is too large , a user can easily observe that SGD diverges in a few iterations and directly stops the current job . Therefore , the random search hypothesis will over-emphasize the role of hyperparameter tuning and does not align with a real user ’ s practical efficiency . Second , the performance of the best hyperparameter is crucial for many applications . For example , in many real-world applications , we need to re-train the model every day or every week with newly added data . So the best hyperparameter selected in the beginning might benefit all these re-train tasks rather than searching parameters from scratch . In addition , due to the expensive random search , random search based evaluation often focuses on the low-accuracy region2 , while practically we care about the performance for getting reasonably good accuracy . Our contributions Given that hyperparameter tuning is either under-emphasized ( assuming the best hyperparameters ) or over-emphasize ( assuming random search ) in existing benchmarking protocols and comparisons , we develop new evaluation protocols to compare optimizers to reflect the real use cases better . Our evaluation framework includes two protocols . First , to evaluate the end-to-end training efficiency for a user to train the best model from scratch , we develop an efficient evaluation protocol to compare the accuracy obtained under various time budgets , including the hyperparameter tuning time . Instead of using the random search algorithm , we adopt the Hyperband ( Li et al. , 2017 ) algorithm for hyperparameter tuning since it can stop early for bad configurations and better reflect the real running time required by a user . Further , we also propose to evaluate the data addition training efficiency for a user to re-train the model with some newly added training data , with the knowledge of the best hyperparameter tuned in the previous training set . We also conduct human studies to study how machine learning researchers are tuning parameters in optimizers and how that aligns with our proposed protocols . Based on the proposed evaluation protocols , we study how much progress has recently proposed algorithms made compared with SGD or Adam . Note that most of the recent proposed optimizers have shown outperforming SGD and Adam under the best hyperparameters for some particular tasks , but it is not clear whether the improvements are still significant when considering hyper-parameter tuning , and across various tasks . To this end , we conduct comprehensive experiments comparing state-of-the-art training algorithms , including SGD ( Robbins & Monro , 1951 ) , Adam ( Kingma & Ba , 2014 ) , RAdam ( Liu et al. , 2019 ) , Yogi ( Zaheer et al. , 2018 ) , LARS ( You et al. , 2017 ) , LAMB ( You et al. , 2019 ) , and Lookahead ( Zhang et al. , 2019 ) , on a variety of training tasks including image classification , generated adversarial networks ( GANs ) , sentence classification ( BERT fine-tuning ) , reinforcement learning and graph neural network training . Our main conclusions are : 1 ) On CIFAR-10 and CIFAR-100 , all the optimizers including SGD are competitive . 2 ) Adaptive methods are generally better on more complex tasks ( NLP , GCN , RL ) . 3 ) There is no clear winner among adaptive methods . Although RAdam is more stable than Adam across tasks , Adam is still a very competitive baseline even compared with recently proposed methods . 2For instance , Sivaprasad et al . ( 2020 ) only reaches < 50 % accuracy in their CIFAR-100 comparisons . 2 RELATED WORK . Optimizers . Properties of deep learning make it natural to apply stochastic first order methods , such as Stochastic Gradient Descent ( SGD ) ( Robbins & Monro , 1951 ) . Severe issues such as a zig-zag training trajectory and a uniform learning rate have been exposed , and researchers have then drawn extensive attention to modify the existing SGD for improvement . Along this line of work , tremendous progresses have been made including SGDM ( Qian , 1999 ) , Adagrad ( Duchi et al. , 2011 ) , RMSProp ( Tieleman & Hinton , 2012 ) , and Adam ( Kingma & Ba , 2014 ) . These methods utilize momentums to stabilize and speed up training procedures . Particularly , Adam is regarded as the default algorithm due to its outstanding compatibility . Then variants such as Amsgrad ( Reddi et al. , 2019 ) , Adabound ( Luo et al. , 2019 ) , Yogi ( Zaheer et al. , 2018 ) , and RAdam ( Liu et al. , 2019 ) have been proposed to resolve different drawbacks of Adam . Meanwhile , the requirement of large batch training has inspired the development of LARS ( You et al. , 2017 ) and LAMB ( You et al. , 2019 ) . Moreover , Zhang et al . ( 2019 ) has put forward a framework called Lookahead to boost optimization performance by iteratively updating two sets of weights . Hyperparameter tuning methods . Random search and grid search ( Bergstra & Bengio , 2012 ) can be a basic hyperparameter tuning method in the literature . However , the inefficiency of these methods stimulates the development of more advanced search strategies . Bayesian optimization methods including Bergstra et al . ( 2011 ) and Hutter et al . ( 2011 ) accelerate random search by fitting a black-box function of hyperparameter and the expected objective to adaptively guide the search direction . Parallel to this line of work , Hyperband ( Li et al. , 2017 ) focuses on reducing evaluation cost for each configuration and early terminates relatively worse trials . Falkner et al . ( 2018 ) proposes BOHB to combine the benefits of both Bayesian Optimization and Hyperband . All these methods still require huge computation resources . A recent work ( Metz et al. , 2020 ) has tried to obtain a list of potential hyperparameters by meta-learning from thousands of representative tasks . We strike a balance between effectiveness and computing cost and leverage Hyperband in our evaluation protocol to compare a wider range of optimizers . 3 PROPOSED EVALUATION PROTOCOLS . In this section , we introduce the proposed evaluation framework for optimizers . We consider two evaluation protocols , each corresponding to an important training scenario : • Scenario I ( End-to-end training ) : This is the general training scenario , where a user is given an unfamiliar optimizer and task , the goal is to achieve the best validation performance after several trials and errors . In this case , the evaluation needs to include hyperparameter tuning time . We develop an efficiency evaluation protocol to compare various optimizers in terms of CPE and peak performance . • Scenario II ( Data-addition training ) : This is another useful scenario encountered in many applications , where the same model needs to be retrained regularly after collecting some fresh data . In this case , a naive solution is to reuse the previously optimal hyperparameters and retrain the model . However , since the distribution is shifted , the result depends on the sensitivity to that shift . We describe the detailed evaluation protocol for each setting in the following subsections . 3.1 END-TO-END TRAINING EVALUATION PROTOCOL . Before introducing our evaluation protocol for Scenario I , we first formally define the concept of optimizer and its hyperparameters . Definition 1 . An optimizer is employed to solve a minimization problem minθ L ( θ ) and can be defined by a tuple o ∈ O = ( U , Ω ) , where O contains all types of optimizers . U is a specific update rule and Ω = ( ω1 , . . . , ωN ) ∈ RN represents a vector ofN hyperparameters . Search space of these hyperparameters is denoted by F . Given an initial parameter value θ0 , together with a trajectory of optimization procedure Ht = { θs , L ( θs ) , ∇L ( θs ) } , the optimizer updates θ by θt+1 = U ( Ht , Ω ) . We aim to evaluate the end-to-end time for a user to get the best model , including the hyperparameter tuning time . A recent work ( Sivaprasad et al. , 2020 ) assumes that a user conducts random search for finding the best hyperparameter setting . Still , we argue that the random search procedure will over-emphasize the importance of hyperparameters when tuning is considered — it assumes a user never stops the training even if they observe divergence or bad results in the initial training phase , which is unrealistic . Figure 1 illustrates why random search might not lead to a fair comparison of optimizers . In Figure 1 , we are given two optimizers , A and B , and their corresponding loss w.r.t . hyperparameter . According to Sivaprasad et al . ( 2020 ) , optimizer B is considered better than optimizer A under a constrained budget since most regions of the hyperparameter space of A outperforms B . For instance , suppose we randomly sample the same hyperparamter setting for A and B . The final config ω∗r ( B ) found under this strategy can have a lower expected loss than that of ω∗r ( A ) , as shown in Figure 1a . However , there exists a more practical search strategy which can invalidate this statement with the assumption of a limited searching budget : a user can early terminate a configuration trial when trapped in bad results or diverging . Hence , we can observe in Figure 1b that for optimizer A , this strategy earlystops many configurations and only allow a limited number of trials to explore to the deeper stage . Therefore , the bad hyperparameters will not affect the overall efficiency of Optimizer A too much . In contrast , for optimizer B , performances of different hyperparameters are relatively satisfactory and hard to distinguish , resulting in similar and long termination time for each trial . Therefore , it may be easier for a practical search strategy p to find the best configuration ω∗p ( A ) of optimizer A than ω∗p ( B ) , given the same constrained budget . This example suggests that random search may over-emphasize the parameter sensitivity when benchmarking optimizers . To better reflect a practical hyperparameter tuning scenario , our evaluation assumes a user applies Hyperband ( Li et al. , 2017 ) , a simple but effective hyperparameter tuning scheme to get the best model . Hyperband formulates hyperparameter optimization as a unique bandit problem . It accelerates random search through adaptive resource allocation and earlystopping , as demonstrated in Figure 1b . Compared with its more complicated counterparts such as BOHB ( Falkner et al. , 2018 ) , Hyperband requires less computing resources and performs similarly within a constrained budget . The algorithm is presented in Appendix A . Despite different hyperparameter tuning algorithms , human tuning by experts is still regarded as the most effective . To verify and support that Hyperband is an effective method and even competitive with humans , we conduct a human study as follows : for image classification on CIFAR10 , given 10 learning rate configurations of SGD in the grid [ 1.0 × 10−8 , 1.0 × 10−7 , 1.0 × 10−6 , . . . , 10 ] , participants are requested to search the best one at their discretion . Namely , they can stop or pause a trial any time and continue to evaluate a new configuration until they feel it has already reached the best performance . 10 participants are sampled randomly from Ph.D. students with computer science backgrounds . We collect their tuning trajectories and average them as human performance , which is considered “ optimal ” in this human study . In Figure 2 , we plot curves for hyperparameter tuning of human , Hyperband , random search , random search with an early stopping ( ES ) strategy in Sivaprasad et al . ( 2020 ) , and Hyperband with ES . We find that Hyperband matches humans ’ behavior better , while random search tends to trap in suboptimal configurations although random with early stopping can mitigate this issue to some extent . This finding shows the advantage of Hyperband over random search regardless of early stopping , and justifies the use of Hyperband in optimizer benchmarking . More details of this human study can be found in Appedix B . With Hyperband incorporated in end-to-end training , we assume that each configuration is run sequentially and record the best performance obtained at time step t as Pt . Specifically , Pt represents the evaluation metric for each task , e.g. , accuracy for image classification and return for reinforcement learning . { Pt } Tt=1 forms a trajectory for plotting learning curves on test set like Figure 3 . Although it is intuitive to observe the performance of different optimizers according to such figures , summarizing a learning curve into a quantifiable , scalar value can be more insightful for evaluation . Thus , as shown in Eq . 1 , we use λ-tunability defined in Sivaprasad et al . ( 2020 ) to further measure the performance of optimizers : λ-tunability = ∑T t=1 λt · Pt , where ∑ t λt = 1 and ∀tλt > 0 . ( 1 ) One intuitive way is to set λt = 1t=T to determine which optimizer can reach the best model performance after the whole training procedure . However , merely considering the peak performance is not a good guidance on the choice of optimizers . In practice , we tend to take into account the complete trajectory and exert more emphasis on the early stage . Thus , we employ the Cumulative Performance-Early weighting scheme where λt ∝ ( T − i ) , to compute λ-tunablity instead of the extreme assignment λt = 1t=T . The value obtained is termed as CPE for simplicity . We present our evaluation protocol in Algorithm 1 . As we can see , end-to-end training with hyperparameter optimization is conducted for various optimizers on the given task . The trajectory { Pt } Tt=1 is recorded to compute the peak performance as well as CPE value . Note that the procedure is repeated M times to obtain a reliable result . We use M = 3 in all experiments . More details of time cost and acceleration of the algorithm can be found in Appendix E. Algorithm 1 End-to-End Efficiency Evaluation Protocol Input : A set of optimizers O = { o : o = ( U , Ω ) } , task a ∈ A , feasible search space F 1 : for o ∈ O do 2 : for i = 1 to M do 3 : Conduct hyperparameter search in F with the optimizer o using HyperBand on a 4 : Record the performance trajectory { Pt } Tt=1 explored by HyperBand 5 : Calculate the peak performance and CPE by Eq . 1 6 : end for 7 : Average peak and CPE values over M repetitions for the optimizer o 8 : end for 9 : Evaluate optimizers according to their peak and CPE values

Summary: This paper studies the topic of evaluating the performance of optimizers for neural networks. The paper makes the argument that existing evaluation procedures either over emphasize the finding of optimal hyperparameters or under-evaluate the performance of an algorithm by randomly sampling hyperparameters. This paper's primary objective is to propose an evaluation procedure that better aligns with a practitioner's goal than existing evaluation procedures. The proposed procedure evaluates optimization algorithms by using the hyperband hyperparameter optimization algorithm to tune hyperparameters and then score the algorithm using a weighted combination of validation performance scores over regularly sampled training intervals. The aggregate performances of algorithms are then ranked using performance profiles.