Massively Parallel Hyperparameter Tuning

1 INTRODUCTION . Although machine learning models have recently achieved dramatic successes in a variety of practical applications , these models are highly sensitive to internal parameters , i.e. , hyperparameters . In these modern learning regimes , three trends motivate a new approach to hyperparameter tuning : 1 . High-dimensional hyperparameter spaces . Machine learning models are becoming increasingly complex , as evidenced by modern neural networks with dozens of hyperparameters . For such complex models with hyperparameters that interact in unknown ways , a practitioner is forced to evaluate potentially thousands of different hyperparameter settings . 2 . Increasing training times . As datasets grow larger and models become more complex , training a model has become dramatically more expensive , often taking days or weeks on specialized high-performance hardware . This trend is particularly onerous in the context of hyperparameter tuning , as a new model must be trained to evaluate each candidate hyperparameter configuration . 3 . Rise of parallel computing . The combination of a growing number of hyperparameters and longer training time per model precludes evaluating configurations sequentially ; we simply can not wait months or years to find a suitable hyperparameter setting . Leveraging parallel and distributed computational resources presents a solution to the increasingly challenging problem of hyperparameter optimization . Our goal is to design a hyperparameter tuning algorithm that can effectively leverage parallel resources . This goal seems trivial with standard methods like random search , where we could train different configurations in an embarrassingly parallel fashion . However , for high-dimensional search spaces , the number of candidate configurations required to find a good configuration often dwarfs the number of available parallel resources , and when possible , our goal is to : Evaluate orders of magnitude more hyperparameter configurations than available parallel workers in a small multiple of the wall-clock time needed to train a single model . We denote this setting as the large-scale regime for hyperparameter optimization . In this work we study this regime , and our contributions are as follows : • We introduce the Asynchronous Successive Halving Algorithm ( ASHA ) , a practical hyperparameter tuning method for the large-scale regime that exploits parallelism and aggressive early-stopping . Our algorithm is inspired by the Successive Halving algorithm ( SHA ) ( Karnin et al. , 2013 ; Jamieson & Talwalkar , 2015 ) , a theoretically principled early-stopping method that allocates more resources to promising configurations . As a side benefit of adapting SHA for the parallel setting , we also improve upon the synchronous version of SHA , as described in Section 3.3 . • In the sequential setting , we compare SHA with BOHB ( Falkner et al. , 2018 ) , Fabolas ( Klein et al. , 2017 ) , and Population Based Tuning ( PBT ) ( Jaderberg et al. , 2017 ) , state-of-the-art methods that exploit partial training . Our results show that SHA slightly outperforms these methods , which when coupled with SHA ’ s simplicity and theoretical grounding , motivate the use of SHA in parallel settings . We also verify that SHA and ASHA achieve similar results . • In a distributed setting with 25 workers , we evaluate ASHA ’ s suitability for the large-scale regime . We focus on tuning two CNNs on CIFAR-10 , and observe that ASHA finds a good configuration in approximately the time it takes to train a single model . We also show that ASHA scales linearly with the number of workers , and that it again slightly exceeds the performance of PBT and BOHB , a method combining Bayesian optimization with SHA . • In a distributed setting with 500 workers , we compare ASHA to the default tuning algorithm used by Vizier ( Golovin et al. , 2017 ) , Google ’ s internal hyperparameter tuning service . We show that ASHA outperforms Vizier when tuning an LSTM model on the Penn Treebank dataset ( PTB ) . • Finally , we compare ASHA and PBT on a task using 16 GPUs to tune a near state-of-the-art language model ( Merity et al. , 2018 ) , with a published test perplexity of 57.3 on PTB . ASHA outperforms PBT and is able to find a configuration with a test perplexity of 56.3 in less than 2× the time it takes to train a single model . 2 RELATED WORK . Sequential methods : There is a large body of work in this area , with existing methods focusing on adaptively evaluating configurations , adaptively selecting configurations , or taking a hybrid approach that combines both strategies . SHA ( Jamieson & Talwalkar , 2015 ) and Hyperband ( Li et al. , 2018 ) are empirically effective adaptive evaluation techniques that employ principled ‘ early-stopping. ’ SHA serves as the inner loop for Hyperband , with Hyperband automating the choice of the early-stopping rate by running different variants of SHA . While the appropriate choice of early stopping rate is problem dependent , Li et al . ( 2018 ) ’ s empirical results show that aggressive early-stopping works well for a wide variety of tuning tasks . Hence , we focus on adapting SHA to the parallel setting in Section 3 , though we also evaluate the corresponding asynchronous Hyperband method . Fabolas ( Klein et al. , 2017 ) , a hybrid Bayesian optimization method , has demonstrated state-of-theart performance on several tasks in comparison to Hyperband and other leading methods . While Fabolas , along with most other Bayesian optimization approaches , can be parallelized using a constant liar ( CL ) type heuristic ( Ginsbourger et al. , 2010 ; González et al. , 2016 ) , the parallel version will underperform the sequential version , since the latter uses a more accurate posterior to propose new points . Hence , we compare to Fabolas in the sequential setting in Section 4.1 , and demonstrate that under an appropriate experimental setup , SHA and Hyperband in fact slightly outperform Fabolas . More recently in parallel work , Falkner et al . ( 2018 ) introduced BOHB , a hybrid method combining Bayesian optimization with Hyperband . They also propose a parallelization scheme for SHA that retains synchronized eliminations of underperforming configurations . We discuss the drawbacks of this parallelization scheme in Section 3 and demonstrate that ASHA outperforms this version of parallel SHA as well as BOHB in Section 4.2 . Parallel methods : Jaderberg et al . ( 2017 ) developed PBT , a state-of-the-art hybrid evolutionary approach that exploits partial training to iteratively increase the fitness of a population of models . Additionally , Golovin et al . ( 2017 ) introduced Vizier , a black-box optimization service with support for multiple tuning methods and early-stopping options . For succinctness , we will refer to Vizier ’ s default algorithm as “ Vizier ” with the understanding that it is simply one of methods available on the Vizier platform . We compare to PBT and Vizier in Section 4.2 and Section 4.3 , respectively . We further note that PBT is more comparable to SHA than to Hyperband since both PBT and SHA require the user to set the early-stopping rate via internal hyperparameters . Neural Architecture Search ( NAS ) methods : Another class of related methods are tailored for search spaces designed to tune the architectures of neural networks . While earlier methods were computationally expensive ( Zoph & Le , 2017 ; Real et al. , 2017 ) , more recent methods exploit reuse to reduce the costs ( Pham et al. , 2018 ; Bender et al. , 2018 ) . However , these methods suffer from reproducibility issues ( Pham et al. , 2018 ; Pham & Guan , 2018 ) , and recent work has demonstrated that appropriately tuning more traditional architectures can match or outperform the results from architecture search ( Melis et al. , 2018 ) . Relatedly , ASHA ’ s performance in Section 4.3.1 matches that of leading NAS methods on the PTB benchmark ( Pham et al. , 2018 ; Pham & Guan , 2018 ) . 3 ASHA ALGORITHM . We start with an overview of SHA ( Karnin et al. , 2013 ; Jamieson & Talwalkar , 2015 ) and motivate the need to adapt it to the parallel setting . Then we present ASHA and discuss how it addresses issues with synchronous SHA and improves upon the original algorithm . 3.1 SUCCESSIVE HALVING ( SHA ) . Algorithm 1 : Successive Halving Algorithm . 1 Input : number of configurations n , minimum resource r , maximum resource R , reduction factor η , minimum early-stopping rate s 2 smax = blogη ( R/r ) c 3 assert n ≥ ηsmax−s so that at least one configuration will be allocated R. 4 T = get_hyperparameter_configuration ( n ) 5 for i ∈ { 0 , . . . , smax − s } do 6 // All configurations trained for a given i constitute a “ rung. ” 7 ni = bnη−ic 8 ri = rη i+s 9 L = run_then_return_val_loss ( θ , ri ) : θ ∈ T 10 T = top_k ( T , L , ni/η ) 11 end 12 return best configuration in T The idea behind SHA ( Algorithm 1 ) is simple : allocate a small budget to each configuration , evaluate all configurations and keep the top 1/η , increase the budget per configuration by a factor of η , and repeat until the maximum per-configuration budget of R is reached ( lines 5–11 ) . The resource allocated by SHA can be iterations of stochastic gradient descent , number of training examples , number of random features , etc . SHA requires the number of configurations n , a minimum resource r , a maximum resource R , a reduction factor η ≥ 2 , and a minimum early-stopping rate s. Additionally , the get_hyperparameter_configuration ( n ) subroutine returns n configurations sampled randomly from a given search space ; and the run_then_return_val_loss ( θ , r ) subroutine returns the validation loss after training the model with the hyperparameter setting θ and for r resources . For a given early-stopping rate s , a minimum resource of r0 = rηs will be allocated to each configuration . Hence , lower s corresponds to more aggressive early-stopping , with s = 0 prescribing a minimum resource of r. We will refer to SHA with different values of s as brackets and , within a bracket , we will refer to each round of promotion as a rung with the base rung numbered 0 and increasing . Figure 1 ( left ) shows the rungs for bracket 0 for an example setting with n = 9 , r = 1 , R = 9 , and η = 3 , while Figure 1 ( right ) shows how resource allocations change for different brackets . The naive way of parallelizing SHA is to distribute the training of the n/ηk surviving configurations on each rung k as is done by Falkner et al . ( 2018 ) and add brackets when there are no jobs available in existing brackets . We will refer to this method as “ synchronous ” SHA . The efficacy of this strategy is severely hampered by two issues : ( 1 ) SHA ’ s synchronous nature is sensitive to stragglers and dropped jobs as every configuration within a rung must complete before proceeding to the next rung , and ( 2 ) the estimate of the top 1/η configurations for a given early-stopping rate does not improve as more brackets are run since promotions are performed independently for each bracket . We demonstrate in Appendix A.1 that ASHA is more robust than synchronous SHA to stragglers and dropped jobs using simulated workloads . Nonetheless , we compare synchronous SHA in Section 4.3 and show that ASHA performs better . We could also consider the embarrassingly parallel approach of running multiple instances of SHA , one on each worker . However , this strategy is not well suited for the large-scale regime , where we would like results in little more than the time to train one configuration . To see this , assume that training time for a configuration scales linearly with the allocated resource and time ( R ) represents the time required to train a configuration for the maximum resource R. In general , for a given bracket s , the minimum time to return a configuration trained to completion is ( logη ( R/r ) −s+1 ) ×time ( R ) , where logη ( R/r ) − s+ 1 counts the number of rungs . For example , consider Bracket 0 in the toy example in Figure 1 . The time needed to return a fully trained configuration is 3× time ( R ) , since there are three rungs and each rung is allocated R resource . Moreover , using this parallelization strategy on the task considered in Section 4.3.1 would have required 5 days to return an answer , compared to the 1 day needed for ASHA . Algorithm 2 : Asynchronous Successive Halving Algorithm . 1 Input : minimum resource r , maximum resource R , reduction factor η , minimum early-stopping rate s 2 Algorithm ASHA ( ) 3 repeat 4 for each free worker do 5 ( θ , k ) = get_job ( ) 6 run_then_return_val_loss ( θ , rηs+k ) 7 end 8 for completed job ( θ , k ) with loss l do 9 Update configuration θ in rung k with loss l. 10 end 11 Procedure get_job ( ) 12 // Check to see if there is a promotable config . 13 for k = blogη ( R/r ) c − s , . . . , 1 , 0 do 14 candidates = top_k ( rung k , |rung k|/η ) 15 promotable = { t for t ∈ candidates if t not already promoted } 16 if |promotable| > 0 then 17 return promotable [ 0 ] , k + 1 18 end 19 end 20 Draw random configuration θ . // If not , grow bottom rung . 21 return θ , 0

Summary: The paper describes a simple, asynchronous way to parallelize successive halving. In a nutshell, this method, dubbed ASHA, promotes a hyperparameter configuration to the next rung of successive halving when ever possible, instead of waiting that all configurations of the current rung have finished. ASHA can easily be combined with Hyperband which iteratively runs different brackets of successive halving. The empirical evaluation shows that the proposed method outperforms in the most cases other parallel methods such as Vizier and population based training.