Model-based Asynchronous Hyperparameter and Neural Architecture Search

1 INTRODUCTION . The goal of hyperparameter and neural architecture search ( HNAS ) is to automate the process of finding the right architecture or hyperparameters x ? ∈ argminx∈X f ( x ) of a deep neural network by minimizing the validation loss f ( x ) , observed through noise : yi = f ( xi ) + i , i ∼ N ( 0 , σ2 ) , i = 1 , . . . , n. Bayesian optimization ( BO ) is an effective model-based approach for solving expensive black-box optimization problems ( Jones et al. , 1998 ; Shahriari et al. , 2016 ) . It constructs a probabilistic surrogate model of the loss function p ( f | D ) based on previous evaluations D = { ( xi , yi ) } ni=1 . Searching for the global minimum of f is then driven by trading off exploration in regions of uncertainty and exploitation in regions where the global optimum is likely to reside . However , for HNAS problems , standard BO needs to be augmented in order to remain competitive . For example , training runs for unpromising configurations x can be stopped early , but serve as low-fidelity approximations of f ( x ) ( Swersky et al. , 2014 ; Domhan et al. , 2015 ; Klein et al. , 2017b ) . Further , evaluations of f can be executed in parallel to reduce the wall-clock time required for finding a good solution . Several methods for multi-fidelity and/or distributed BO have been proposed ( Kandasamy et al. , 2017 ; 2016 ; Takeno et al. , 2020 ) , but they rely on rather complicated approximations to either select the fidelity level or to compute an information theoretic acquisition function to determine the next candidate . In this work we aim to adopt the desiderata of Falkner et al . ( 2018 ) , namely , that of simplicity , which often leads to more robust methods in practice . A simple , easily parallelizable multi-fidelity scheduling algorithm is successive halving ( SH ) ( Karnin et al. , 2013 ; Jamieson & Talwalkar , 2016 ) , which iteratively eliminates poorly performing neural networks over time . Hyperband ( Li et al. , 2017 ) iterates over multiple rounds of SH with varying ratios between the number of configurations and the minimum amount of resources spent per configuration . Falkner et al . ( 2018 ) introduced a hybrid model , called BOHB , that uses a probabilistic model to guide the search while retaining the efficient any-time performance of Hyperband . However , both SH and BOHB can be bottlenecked due to their synchronous nature : stopping decisions are done only after synchronizing all training jobs at certain resource levels ( called rungs ) . This approach is wasteful when the evaluation of some configurations take longer than others , as it is often the case when training neural networks ( Ying et al. , 2019 ) , and can substantially delay progress towards high-performing configurations ( see example shown in Figure 1 ) . Recently , Li et al . ( 2018 ) proposed ASHA , which adapts successive halving to the asynchronous parallel case . Even though ASHA only relies on the random sampling of new configurations , it has been shown to outperform synchronous SH and BOHB . In this work , we augment ASHA with a Gaussian process ( GP ) surrogate , which jointly models the performance across configurations and rungs to improve the already strong performance of ASHA . The asynchronous nature further requires to handle pending evaluations in principled way to obtain an efficient model-based searcher . 1.1 CONTRIBUTIONS . Dealing with hyperparameter optimization ( HPO ) in general , and HNAS for neural networks in particular , we would like to make the most efficient use of a given parallel computation budget ( e.g. , number of compute instances ) in order to find high-accuracy solutions in the shortest wall-clock time possible . Besides exploiting low-fidelity approximations , we demonstrate that asynchronous parallel scheduling is a decisive factor in cost-efficient search . We further handle pending evaluations through fantasizing ( Snoek et al. , 2012 ) , which is critical for asynchronous searchers . Our novel combination of asynchronous SH with multi-fidelity BO , dubbed MOdel Based aSynchronous mulTi fidelity optimizER ( MOBSTER ) substantially outperforms either of them in isolation . More specifically : • We clarify differences between existing asynchronous SH extensions : ASHA , as described by Li et al . ( 2018 ) and an arguably simpler stopping rule variant , related to the median rule ( Golovin et al. , 2017 ) and first implemented in Ray Tune ( Liaw et al. , 2018 ) . Although their difference seem subtle , they lead to substantially different behaviour in practice . • We present an extensive ablation study , comparing and analysing different components for asynchronous HNAS . While these individual components are not novel , one of our main contributions is to show that , by systematically combining them to form MOBSTER , we obtain a reliable and more efficient method than other recently proposed approaches . Due to limited space , we present a detailed description of the technical nuances and complexities of model-based asynchronous multi-fidelity HNAS in Appendix A.3 . • On a variety of neural network benchmarks , we show that MOBSTER is more efficient in terms of wall-clock time than other state-of-the-art algorithms . Unlike BOHB , it does not suffer from substantial synchronization overheads when evaluations are expensive . As a result , we can achieve the same performance in the same amount of wall-clock time but with often just half the computational resources compared to random-sampling based ASHA . Next , we relate our work to approaches recently published in the literature . In Section 2 , we review synchronous SH , as well as two asynchronous extensions . Our novel method is presented in Section 3 . We present empirical evaluations for the HNAS of various neural architecture types in Section 4 , and finish with conclusions and future work in Section 5 . 1.2 RELATED WORK . A range of prior Gaussian process ( GP ) based BO work exploits multiple fidelities of the objective ( Kennedy & O ’ Hagan , 2000 ; Klein et al. , 2017a ; Poloczek et al. , 2018 ; Cutajar et al. , 2018 ) . A joint GP model across configurations and tasks was used by Swersky et al . ( 2013 ) , allowing for trade-offs between cheap auxiliaries and the expensive target . Klein et al . ( 2017a ) presented a continuous multi-fidelity BO method , where training set size is an input . It relies on a complicated and expensive acquisition function , and experiments with asynchronous scheduling are not provided . Kandasamy et al . ( 2017 ) presented BOCA , a Bayesian optimization method that exploits general continuous fidelities of the objective function . While BOCA uses a model similar to ours , they employ a different strategy to select the fidelity , which appears to work less efficiently in practice , as we show in our experiments . The “ freeze-thaw ” approach ( Swersky et al. , 2014 ) allows for asynchronous parallel evaluations , yet its scheduling is quite different from asynchronous SH . They propose an exponential decay kernel , a refinement of which we use here . No results for deep neural network tuning were presented , and no implementation is publicly available . Previous work on asynchronous BO has demonstrated performance gains over synchronous BO methods ( Alvin et al. , 2019 ; Kandasamy et al. , 2016 ) , yet this work did not exploit multiple fidelities of the objective . Concurrently to our work , Takeno et al . ( 2020 ) proposed an asynchronous multi-fidelity BO method . However , it is based on an information-theoretic acquisition function that is substantially more complex than our method . Since no implementation is publicly available it is omitted from our comparisons . Falkner et al . ( 2018 ) combine synchronous Hyperband with TPE ( Bergstra et al. , 2011 ) as a model . Their method , called BOHB , combines early speed-ups of Hyperband with fast convergence later on , and shows competitive performance for NAS ( Ying et al. , 2019 ; Dong & Yang , 2020 ) . However , compared to our approach , it fits independent kernel density estimator at each rung level , which prevents the interpolation to higher rung levels . Apart from this , kernel density estimators do not permit fantasizing the potential outcome of a pending configuration , which makes it unclear how to extend BOHB to the asynchronous setting , and are notoriously sensitive to their bandwidth parameters . 2 SYNCHRONOUS AND ASYNCHRONOUS MULTI-FIDELITY SCHEDULING . Multi-fidelity HNAS considers an objective f ( x , r ) , indexed by the resource level r ∈ { rmin , . . . , rmax } . The target function of interest is f ( x ) = f ( x , rmax ) , while evaluations of f ( x , r ) for r < rmax are cheaper , lower-fidelity evaluations that may be correlated with f ( x ) . Here , we consider the number of training epochs ( i.e. , full sweeps over the data ) as the resource r , though other choices are possible , such as training subset ratios ( Klein et al. , 2017a ) . Two kinds of decisions are made in multi-fidelity HNAS : choosing the configurations to evaluate and scheduling their evaluations . In this work , configurations are chosen non-uniformly based on the GP surrogate model that accounts for pending and actual evaluations . Scheduling decisions are stop/go decisions that happen at certain resource levels called rungs . We make use of successive halving ( SH ) for its simplicity and strong empirical performance ( Jamieson & Talwalkar , 2016 ; Li et al. , 2018 ) . Let η ∈ { 2 , 3 , 4 } be the halving constant , rmax and rmin the maximum and minimum resource level for an evaluation . We assume , for simplicity , that rmax/rmin = ηK , where K ∈ N. The full set of rungs is R = { rminηk | k = 0 , . . . , K } . In the sequel , the term “ rung ” will be overloaded , denoting both the resource level at which stop/go decisions are made and the list of configurations that reached it . In synchronous SH , each rung has an a priori fixed size ( i.e. , number of slots for configurations evaluated until the rung level ) , the size ratio between successive rungs being η−1 ( e.g. , n ( r ) = rmax/r for r ∈ R ) . A round of the algorithm starts with evaluating n ( rmin ) configurations to the lowest rung r = rmin , making use of parallel computation if available . Once all of them finish , the η−1 fraction of top performing configurations are promoted to the next rung while the others are terminated . Each rung is a synchronization point for many training tasks : it has to be populated entirely before any configuration can be promoted ( see Figure 1 left for a visualization ) . Synchronous scheduling makes less efficient use of parallel computation than asynchronous scheduling . Configurations in the same rung may require quite different compute times , for example , if we search over hyperparameters that control the network size . Hence , some workers may just be idle at a synchronization point . Evaluations at larger resource levels are observed earlier with asynchronous scheduling . However , the risk is to continue mediocre configurations because they were selected earlier . Figure 1 illustrates the differences between asynchronous and synchronous scheduling . We distinguish two variants of asynchronous SH proposed in prior work . During the optimization process , for any hyperparameter and architecture configuration x ∈ X that is currently being evaluated , a decision needs to be made once it reaches the next rung r ∈ R. Given the new observed data point y at level r , the binary predicate continue ( x , r , y ) evaluates to true iff y is in the top η−1 fraction of records at the rung . Stopping variant . This is a simple extension of the median stopping rule ( Golovin et al. , 2017 ) , which is implemented in Ray Tune ( Liaw et al. , 2018 ) . As soon as a job for x reaches a rung r , if continue ( x , r , y ) is true , it continues towards the next rung level . Otherwise , it is stopped , and the worker becomes free to pick up a novel evaluation . As long as fewer than η configurations have reached a rung , the job is always continued . Promotion variant . This variant of asynchronous SH was presented by Li et al . ( 2018 ) and called ASHA . Note that , in the remaining text , we will refer to general asynchronous SH as ASHA and indicate the both variants with promotion or stopping based ASHA . Once a job for some x reaches a rung r , it is paused there , and the worker is released . The evaluation of x can be promoted ( i.e. , continued to the next rung ) later on . When a worker becomes available , rungs are scanned in descending order , running a job to promote the first paused x for which continue ( x , r , y ) is true . When no such paused configuration exists , a new configuration evaluation is started . With fewer than η metrics recorded at a rung , no promotions happen there . The stopping and promotion variants can exhibit a rather different behaviour ( see Figure 2 ) , as also demonstrated in our experiments . Initially , the stopping variant gives most configurations the benefit of doubt , while the promotion variant pauses evaluations until they can be compared against a sufficient number of competitors . Finally , note that asynchronous SH can be generalized to asynchronous Hyperband in much the same way as in the synchronous case , as shown by Li et al . ( 2018 ) . However , it was reported that asynchronous SH typically outperforms the Hyperband for expensive tuning problems , which we also show in our experiments in Section 4.1 . After some initial exploration , both with random and model-based variants , we confirmed this observation . In the following , we will thus restrict our attention to SH scheduling .

Summary: This paper has proposed to exploit a GP model to represent the correlation between configuration-rung tuples (as is typical in multi-fidelity BO) in asynchronous successive halving (ASHA) (Li et al. 2018), which has resulted in performance improvement over the state of the art, as shown in the experimental results. The experimental results are extensive and compelling. Introducing a GP to model the correlation between configuration-rung tuples in the existing ASHA work offers minimal technical merit though. Can the authors elaborate on whether there is any nontrivial, novel technical challenge with such an integration? This question does not seem to be adequately addressed in the paper.