ECONOMIC HYPERPARAMETER OPTIMIZATION WITH BLENDED SEARCH STRATEGY

1 INTRODUCTION . Hyperparameter optimization ( HPO ) of modern machine learning models is a resource-consuming task , which is unaffordable to individuals or organizations with little resource ( Yang & Shami , 2020 ) . Operating HPO in a low-cost regime has numerous benefits , such as democratizing ML techniques , enabling new applications of ML , which requires frequent low-latency tuning , and reducing the carbon footprint . It is inherently challenging due to the nature of the task : trying a large number of configurations of heterogeneous cost and accuracy in a large search space . The expense can accumulate from multiple sources : either a large number of individually cheap trials or a small number of expensive trials can add up the required resources . There have been multiple attempts to address the efficiency of HPO from different perspectives . Each of them has strengths and limitations . For example , Bayesian optimization ( BO ) ( Brochu et al. , 2010 ) , which is a class of global optimization algorithms , is used to minimize the total number of iterations to reach global optima . However , when the cost of different hyperparameter configurations is heterogeneous , vanilla BO may select a configuration that incurs unnecessarily high cost . As opposed to BO , local search ( LS ) methods ( Wu et al. , 2021 ) are able to control total cost by preventing very expensive trials until necessary , but they may get trapped in local optima . Multi-fidelity methods ( Jamieson & Talwalkar , 2016 ) aim to use cheap proxies to replace some of the expensive trials and approximate the accuracy assessment , but can only be used when such proxies exist . A single search strategy is difficult to meet the generic goal of economical HPO . In this work , we propose a blended search strategy which combines global search and local search strategy such that we can enjoy benefits from both worlds : ( 1 ) global search can ensure the convergence to the global optima when the budget is sufficient ; and ( 2 ) local search methods enable a better control on the cost incurred along the search trajectory . Given a particular global and local search method , our framework , which is named as BlendSearch , combines them according to the following design principles . ( 1 ) Instead of sticking with a particular method for configuration selection , we consider both of the candidate search methods and decide which one to use at each round of the configuration selection . ( 2 ) We use the global search method to help decide the starting points of local search threads . ( 3 ) We use the local search method to intervene the global search method ’ s configuration selection to avoid configurations that may incur unnecessarily large evaluation cost . ( 4 ) We prioritize search instances of both methods according to their performance and efficiency of performance improvement on the fly . Extensive empirical evaluation on the AutoML ∗Equal contribution Benchmark ( Gijsbers et al. , 2019 ) validates the robust performance of our method on a wide variety of datasets . BlendSearch is now publicly available in an open-source AutoML Library1 . 2 BACKGROUND AND RELATED WORK . We first briefly introduce the vanilla Bayesian optimization methods and local search methods , which are among the building blocks of our method . Bayesian optimization is a class of global optimization algorithms which is suitable for optimizing expensive black-box functions . It models the probabilistic distribution of the objective conditioned on the optimization variables . Typical models include Gaussian process ( Snoek et al. , 2012 ) , random forest ( Hutter et al. , 2011 ) , and tree Parzen estimator ( TPE ) ( Bergstra et al. , 2011 ) . In BO methods , an acquisition function is used to determine the next point to evaluate . Two common acquisition functions are the expected improvement ( EI ) ( Bull , 2011 ) over the currently best-observed objective and upper confidence bound ( UCB ) ( Srinivas et al. , 2009 ) . Local search methods are prevalent in the general optimization literature ( Spall et al. , 1992 ; Nesterov & Spokoiny , 2017 ) but less studied in the HPO literature due to the possibility of getting trapped in local optima ( György & Kocsis , 2011 ) . Recent work ( Wu et al. , 2021 ) shows that a local search method FLOW2 can make HPO cost-effective when combined with low-cost initialization and random restart . At each iteration , it samples a pair of vectors ( with opposite directions ) uniformly at random from a unit sphere , the center of which is the best configuration found so far ( a.k.a . incumbent ) and the radius of which is the current stepsize . Expensive configurations are avoided in the beginning as each iteration proposes a configuration near the incumbent . Random restart of the local search is performed once the convergence condition is satisfied . There are several attempts to address the limitations of vanilla BO or local search methods . BOwLS ( BO with local search ) ( Gao et al. , 2020 ) uses a BO model to select the starting point of a local search thread . Each local search thread is run until convergence and the BO model is updated with the start point and the converged loss . Trust region BO ( Eriksson et al. , 2019 ) fits a fixed number of local models and performs a principled global allocation of samples across these models via an implicit bandit approach . It is primarily designed for HPO problems with high-dimensional numerical hyperparamters . Unfortunately , all existing work that tries to combine global search with local search methods does not consider the heterogeneity of evaluation cost incurred along with the search . There are also a lot of attempts in making HPO efficient by speeding up configuration evaluation . Multifidelity optimizations ( Klein et al. , 2017 ; Li et al. , 2017 ; Kandasamy et al. , 2017 ; Falkner et al. , 2018 ; Lu et al. , 2019 ; Li et al. , 2020 ) are proposed for this purpose . They usually require an additional degree of freedom in the problem called ‘ fidelity ’ , to allow performance assessment on a configura- 1https : //github.com/microsoft/FLAML tion with different fidelities . There is surprisingly little prior work for generic cost-effective HPO . Gaussian process with expected improvement per second ( GPEIPS ) ( Snoek et al. , 2012 ) models the evaluation cost using another Gaussian process , and heuristically adds the estimated cost into the acquisition function . It does not always outperform GPEI as the acqusition function can overpenalize good but expensive configurations . 3 BLENDSEARCH . Our framework needs the following information as inputs . • B is the total budget of cost . In this work , we measure the cost by CPU/GPU time . • P is the input HPO problem , which has the following attributes characterizing the problem . – P.X is the search space in which each element x is a d-dimensional hyperparameter configuration . For a non-categorical hyperparameter coordinate i ∈ [ d ] , if different values of xi lead to heterogeneous cost and there is a known value P.xLowCosti corresponding to a low cost , it is considered as a controlled dimension . We use P.D to denote such a set of controlled dimensions . – P.LossFunc ( · ) is the loss function to be minimized in terms of the configurations x ∈ P.X . – P.CostFunc ( · ) is the cost function that outputs the cost incurred when evaluating x . The goal of an HPO algorithm is to minimize the loss P.LossFunc ( x ) with the constraint that the total cost incurred G ( π ) : = ∑ x∈I ( π ) P.CostFunc ( x ) ≤ B , where I ( π ) is the search trajectory of algorithm π . Note that both P.LossFunc ( x ) and P.CostFunc ( x ) are black-box functions meaning that typically the analytic form is not available , and only function values can be observed . In order to distinguish the operation of querying from loss/cost function and the loss/cost observation , we use P.LossFunc ( x ) and P.CostFunc ( x ) to denote the former , and use l ( x ) and c ( x ) to denote the latter . l and c ( omitting x ) are used when there is no ambiguity . • G is the global search method to be used . L is the local search method to be used . L.∆ is the largest stepsize used in local search method L , i.e. , the largest possible change on a hyperparameter value between two consecutive search steps . The overall design of our framework is presented in Figure 2 and Algorithm 1 . The key idea is to maintain a pool of search threads , one of which corresponds to global search and the others local search . The pool starts with one global search thread and gradually adds local search threads as the search goes on . Here a search thread is an instance of a global search or local search method , each with its own search trajectory . At each round , a search thread selector selects one of the search threads from the pool according to a priority metric that reflects the search threads ’ current performance and efficiency of performance improvement . The selected search thread will then be used to propose a configuration to evaluate in this round . When the selected search thread is the global search thread , a config validator first checks whether the proposed configuration is within the ‘ admissible ’ region for evaluation . If not , it uses a backup local search thread instead . A local search thread will be created only when the global search thread proposes a valid config and a certain create thread condition is met , and will be deleted once it converges . Two local search threads will be merged into one if they are close enough . The priority of each search thread is updated after each evaluation round . Algorithm 1 BlendSearch Inputs : HPO problem P , a global search method G , a local search method L , total budget B . 1 : Initialization : Initialize F .S = [ S0 ] where S0 is an instance of G. We denote by F .x0 the initial point of the search . By design , the values of the controlled dimensions of F .x0 are set to be P.xLowCost and those for the other dimensions are proposed by S0 . 2 : while F .c < B do 3 : S̃ , S̃bak ← SelectThread ( F ) 4 : ( x , l , c ) ← SearchEvaluate1Step ( S̃ , F , P ) 5 : if x is invalid then ( x , l , c ) ← SearchEvaluate1Step ( S̃bak , F , P ) 6 : if x is proposed by global search & CreateNewLSCondition is satisfied then 7 : Initialize S = InitializeSearchThread ( L , P , ( x , l , c ) ) 8 : Add the new LS thread S into the pool : F .S← F .S + S 9 : DeleteAndMergeLS ( F ) . Merge or delete existing LS threads when necessary 10 : Update F .Priority 11 : if x is proposed by global search then UpdateGSModel ( S0 , ( x , l , c ) ) For convenience , we use F to denote a collection of framework-level variables : • F .S is the list of search threads maintained in our framework . F .S contains at least one search thread and among them there is one and only one global search thread , i.e. , S0 , in F .S . • F .Priority is a priority dictionary , in which the keys are the search threads in F .S , and the values are the priority of the corresponding search threads . • Bookkeeping information of F : F .l∗ is the best loss achieved among all the search threads and F .c is the total cost consumed in our framework . • F .R is the ‘ admissible ’ region on the controlled dimensions of the current search , which is a hyperrectangle and can be written in the form of F .R : = { [ F .xmini , F .xmaxi ] } i∈D with F .xmini and F .xmaxi denoting the minimum and maximum value along the i-th dimension in F .R respectively . They are initially set as F .xmini = F .xmaxi = P.xLowCosti for all i ∈ D. The ‘ admissible ’ region gradually expands during the search : ( 1 ) it is expanded to cover all the points evaluated by all the search threads and all the points that are possible to be reached by the local search within one search step , as shown in line 7 and 8 of Algorithm 2 ; ( 2 ) it expands if a local search thread converges , as shown in line 3 of Algorithm 7 ( included in Appendix A ) . In the following , we explain the key steps in our algorithm . Step 1 : Search thread selector ( line 3 of Alg 1 ) . In addition to the primary search thread S̃ , SelectThread also outputs a backup search thread S̃bak which is guaranteed to be a local search thread . It is set to be none when there is no local search thread yet in F .S . Specifically , S̃ = arg max S∈F.S F .Priority ( S ) , S̃bak = { arg maxS∈ ( F.S\S0 ) F .Priority ( S ) F .S \ S0 6= ∅ None F .S \ S0 = ∅ , ( 1 ) The design of the priority metric follows the principle of optimism in the face of uncertainty from the multi-armed bandit problem to balance exploitation and exploration ( Lattimore & Szepesvári , 2020 ) . Specifically , linear extrapolation is performed adaptively and locally to calculate the improvement speed of each search thread . The estimated future reward based on such a linear extrapolation provides a first-order upper bound of the ground truth future reward assuming each search thread has a diminishing return , i.e. , the speed of improvement decreases as more resource is spent . Formally , we introduce the following variables and functions for each search thread S ∈ F .S . • Bookkeeping information : S.l1st and S.l2nd are the best loss so far and second best loss before the best loss is achieved . S.c1st and S.c2nd are the total cost taken when S.l1st and S.l2nd are achieved respectively . S.c is the total cost spent in S. S.x1st is the best configuration found so far . • S.s is the performance improvement speed of S. It is calculated as S.s = S.l 2nd−S.l1st S.c−S.c2nd . This formula is only valid when there is at least one improvement . Otherwise , we do not have enough information to estimate the speed of improvement . We set the speed to the highest speed of all the search threads when S.l2nd = S.l1st . It is due to an implicit assumption of diminishing return . • S.xmin and S.xmax are the minimum and maximum value of the i-th dimension of all hyperparameters configurations evaluated in S respectively . • S.CostImp ( · ) is a function whose input is a target loss and output is the anticipated cost for S to generate a better loss than this target . We use the following formula to compute it , which is intuitively using the cost for improvement in the past to estimate that in the future . S.CostImp ( l ) = max { S.c− S.c1st , S.c1st − S.c2nd , 2S.l 1st − l S.s } ( 2 ) Our proposed priority metric is essentially the negative of the projected loss of S : F .Priority ( S ) = − ( S.l1st − S.s× b ) ( 3 ) in which b = min ( maxS∈F.S S.CostImp ( F .l∗ ) , B − F .c ) . maxS∈F.S S.CostImp ( F .l∗ ) can be considered as the resource needed for every S to have a better performance than the currently best performance F .l∗ . Our priority metric estimates the loss of each search thread if such an amount of resource ( restricted by the budget left B − F .c ) is given . By considering both the search threads ’ current performance and potential improvement , it provides a fair trade-off between exploiting the currently-best and exploring the potentially-better choices . Algorithm 2 SearchEvaluate1Step Inputs : HPO problem P , search thread S , and F 1 : if S is None then Construct x as follows : generate the controlled dimensions of x by adding Guassian noises on the corresponding dimensions of F .x0 and for the rest of the dimensions sample uniformly at random from the search space P.X . 2 : else x← S.ProposeConfig ( ) 3 : if S = S0 ( i.e. , S is the global search thread ) & x /∈ F .R then x←invalid 4 : else l , c← P.LossFunc ( x ) , P.CostFunc ( x ) . Evaluate configuration x 5 : if x 6= invalid then 6 : BookKeeping ( S , x , l , c , F ) and update speed S.s , 7 : ∀i ∈ P.D , S.xmini ← min { xi , S.xmini } , F .xmini ← min { S.xmini − L.∆ , F .xmini } , 8 : ∀i ∈ P.D , S.xmaxi ← max { xi , S.xmaxi } , F .xmaxi ← max { S.xmaxi + L.∆ , F .xmaxi } 9 : Outputs : x , l , c Step 2 : Config validator and evaluator ( line 4-5 of Alg 1 ) . After a search thread ( and a backup search thread ) is selected , the next step is to propose the next configuration to try with the chosen search thread ( s ) . Intuitively speaking , we consider generating the next configuration to try primarily according to the selected search thread S̃ whose priority is ranked the highest . But we set a guard rail for the global search thread as it may propose an unnecessarily high-cost configuration . We thus introduce a config validator to validate the configurations proposed by global search according to whether they are within the current admissible region of our framework F .R ( line 3 of Alg 2 ) . A configuration marked as ‘ invalid ’ means that it is considered to be prone to incur unnecessarily high cost and will not be evaluated at this round . In this case , the selected backup search thread will be used to perform another round of SearchEvaluate1Step ( line 5 of Alg 1 ) if it is a valid search thread ( i.e. , not none ) . In the case where the backup thread is none , we generate the new configuration according to line 1 of Alg 2 . The config validator helps avoid potentially high-cost evaluation and thus avoid creating local search threads from high-cost points until necessary . It does not stick to local searches forever because the admissible region F .R gets expanded . Note that according to the definition of F .R , only the controlled dimensions of the hyperparameter configurations are subject to validation check . If needed , a multi-fidelity pruning strategy can be used in this config evaluator component . Multi-fidelity pruning does not necessarily yield better performance . So the adoption of multi-fidelity pruning in BlendSearch is optional . Step 3 : Search thread creator , updater and cleaner ( line 6-11 of Alg 1 ) . If the newly proposed configuration is proposed by global search and it is not marked as ‘ invalid ’ , we consider creating a new local search thread using the proposed configuration as a starting point . To make sure the newly created local search thread is relatively good , we first check whether the proposed configuration ’ s performance is better than at least half of the existing threads ’ performance ( specified in the CreateNewLSCondition ) . If so , a new local search thread will be initialized and added to the active search thread pool S. In DeleteAndMergeLS , we check whether a local search thread has converged according to the convergence condition of the specific local search method . If it is , the search thread will be removed from S. In addition , we also go through all the local search threads to see whether the incumbent of a LS thread is reachable in one step by another LS thread with lower loss ( ref . Appendix A ) . If so , the former LS thread will be deleted . After a configuration proposed by global search is evaluated , the observation tuple ( x , l , c ) is then used to update the model of the global search method through function UpdateGSModel . For example , when the global search method is a Bayesian optimization method , the model is the surrogate model used . Due to page limit , detailed pseudocode for several of the straightforward functions mentioned in our framework are provided in Appendix A , including CreateNewLSCondition , DeleteAndMergeLS , InitializeSearchThread and BookKeeping .

Summary: The proposed BlendedSearch (BS) presents an intuitive next step in the combination of global and local search schemes for hyper-parameter optimization (HPO). Global search schemes are widely used for HPO but can suffer from large HPO times since their vanilla forms do not account for function evaluation costs. Local search schemes are usually not widely used for HPO but seem useful if the goal is to restrict the search to a region of the search space where the function evaluation costs do not grow drastically. The proposed BS interleaves global and local search steps to ensure that the global search does not go into regions of high evaluation costs while also avoiding being stuck in local minima.