Multi-Step Budgeted Bayesian Optimization with Unknown Evaluation Costs

Bayesian optimization ( BO ) is a sample-efficient approach to optimizing costly-toevaluate black-box functions . Most BO methods ignore how evaluation costs may vary over the optimization domain . However , these costs can be highly heterogeneous and are often unknown in advance . This occurs in many practical settings , such as hyperparameter tuning of machine learning algorithms or physics-based simulation optimization . Moreover , those few existing methods that acknowledge cost heterogeneity do not naturally accommodate a budget constraint on the total evaluation cost . This combination of unknown costs and a budget constraint introduces a new dimension to the exploration-exploitation trade-off , where learning about the cost incurs a cost itself . Existing methods do not reason about the various trade-offs of this problem in a principled way , leading often to poor performance . We formalize this claim by proving that the expected improvement and the expected improvement per unit of cost , arguably the two most widely used acquisition functions in practice , can be arbitrarily inferior with respect to the optimal non-myopic policy . To overcome the shortcomings of existing approaches , we propose the budgeted multi-step expected improvement , a non-myopic acquisition function that generalizes classical expected improvement to the setting of heterogeneous and unknown evaluation costs . Finally , we show that our acquisition function outperforms existing methods in a variety of synthetic and real problems . 1 Introduction . Bayesian optimization ( BO ) ( Shahriari et al. , 2016 ; Frazier , 2018 ) is a family of algorithms for optimizing black-box functions that performs well when the number of evaluations is limited ( Snoek et al. , 2012 ; Calandra et al. , 2016 ; Griffiths and Hernández-Lobato , 2020 ) . However , most BO algorithms ignore the fact that the cost of evaluating the black-box objective function may vary substantially across the optimization domain and is often unknown . Problems with this feature arise commonly in practice . For instance , in the context of hyperparameter optimization of machine learning algorithms ( Swersky et al. , 2013 ; Wu et al. , 2020 ) , certain values of hyperparameters such as the learning rate may yield longer training times . Similarly , in materials design and robotics , simulation experiments can take longer for certain parameter configurations ( Field , 1999 ) . Figure 1 illustrates heterogeneity in evaluation costs from benchmark problems used in this paper , which can vary by an order of magnitude . Failing to account for these heterogeneous evaluation costs can lead to evaluating an expensive point when another less expensive one would provide equal benefit towards finding the optimum . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . We consider budgeted BO of a black-box objective function , whose evaluation costs are unknown and possibly heterogeneous across the domain . The goal is to find a point with the largest possible objective value by querying the objective function at a sequence of adaptively chosen points , where the total evaluation cost is subject to a budget constraint ( this cost only affects evaluation , and not a point ’ s quality upon implementation ) . While some existing approaches do address heterogeneous evaluation costs , all do so heuristically , e.g . by maximizing a traditional cost-agnostic acquisition function divided by the cost of evaluation ( Snoek et al. , 2012 ; Poloczek et al. , 2017 ; Wu et al. , 2020 ; Lee et al. , 2020b ) , or by rolling out a heuristic base policy ( Lee et al. , 2021 ) . Importantly , most of these approaches accommodate neither budget constraints nor uncertainty about the cost function as part of the exploration-exploitation trade-off . As we argue theoretically and demonstrate through experiments , this can lead to poor performance . A notable exception is the concurrent work of Lee et al . ( 2021 ) , which introduces a budget-aware non-myopic acquisition function based on rollout of a heuristic base policy . This work appeared while the present paper was under review . Main Contributions . Motivated by the above shortcomings in existing work , we provide a principled approach to budgeted BO with unknown and potentially heterogeneous evaluation costs . Our main contributions are : • We propose a Markov decision process ( MDP ) formulation of the budgeted BO problem with unknown and heterogeneous evaluation costs . Our formulation allows for a random time horizon ( i.e. , the last time before the budget is depleted ) , going beyond the fixed-horizon MDPs formulated in existing work on non-myopic BO . • Budgeted multi-step expected improvement ( B-MS-EI ) , a novel look-ahead acquisition function that generalizes classical expected improvement to the budgeted cost-heterogeneous setting . B-MS-EI can be seen as a principled approximation of the optimal policy of our MDP . • We prove that expected improvement ( EI ) and its cost-normalized variant , two popular existing approaches , can be arbitrarily inferior with respect to the optimal non-myopic policy . • An empirical evaluation on a number of synthetic and real-world experiments demonstrates that B-MS-EI performs favorably with respect to other acquisition functions that are widely-used in settings with heterogeneous costs . The remainder of this work is organized as follows : In Section 2 , we review related work . Our problem setup is formalized in Section 3 . In Section 4 , we introduce B-MS-EI and discuss its efficient maximization via one-shot multi-step Monte Carlo trees . Numerical experiments are presented in Section 5 . Finally , we discuss directions of future work and conclude in Section 6 . 2 Related Work . Our work falls within the BO framework ( Frazier , 2018 ) , whose origins date back to the seminal work of Kushner ( 1964 ) . BO has been successful in a wide range of applications , such as hyperparameter tuning of machine learning algorithms ( Snoek et al. , 2012 ; Wu et al. , 2020 ) , materials design ( Zhang et al. , 2020 ) , drug discovery ( Griffiths and Hernández-Lobato , 2020 ) , and robot locomotion ( Calandra et al. , 2016 ; Wang and Jegelka , 2017 ) . Within the BO literature , the works most closely related to ours are those that acknowledge the existence of costs for evaluating the objective function that are heterogeneous across the search space and aim to devise algorithms that are cost-aware . Much of this work has occurred in the multi-fidelity setting ( Swersky et al. , 2013 ; Kandasamy et al. , 2016 , 2017 ; Poloczek et al. , 2017 ; Song et al. , 2019 ; Wu et al. , 2020 ) , i.e. , where cheaper approximations of the objective function are available . The only exceptions known to us are Snoek et al . ( 2012 ) , Lee et al . ( 2020b ) , and Lee et al . ( 2021 ) , which consider the single-fidelity setting . Snoek et al . ( 2012 ) proposes the expected improvement per unit of cost ( EI-PUC ) , i.e , EI ( x ) /c ( x ) 1 where c ( x ) is the cost of evaluating the objective at x and EI ( x ) is the expected improvement . Lee et al . ( 2020b ) proposes a simple variation called the expected improvement per unit of cost with cost cooling ( EI-PUC-CC ) . EI-PUC-CC is defined by EI ( x ) /c ( x ) ν , where ν is the ratio between the current remaining and initial budgets . The intuition behind the cost exponent is that evaluating points with high cost should be discouraged early in the BO loop ( when ν ≈ 1 ) and accommodated as the budget is consumed and ν decreases to 0 . However , neither EI-PUC nor EI-PUC-CC consider uncertainty in the cost or measure the budget-dependent value of information in a principled way . Work concurrent to ours , Lee et al . ( 2021 ) , also tackles budgeted BO with heterogeneous costs using a non-myopic strategy . However , our work differs in both the model and solution method . Lee et al . ( 2021 ) uses a finite-horizon constrained MDP , while our model is an MDP with a random horizon . We argue that the random horizon formulation is more natural : the formulation of Lee et al . ( 2021 ) requires the addition of a zero reward , zero cost state to accommodate trajectories with a small number of evaluations . Within this formulation , Lee et al . ( 2021 ) proposes a rollout acquisition function with a particular heuristic base policy : h− 1 steps of EI-PUC followed by a last step of EI , where h is the number of look-ahead steps performed . The acquisition function is essentially a single step of policy improvement over the “ EI-PUC followed by EI ” heuristic ( Sutton and Barto , 2018 ) . In contrast , our acquisition function aims to directly approximate the optimal policy . The approach of dividing a cost-agnostic acquisition function by some cost term is widely used in practice for addressing heterogeneity in costs , and is closely related to the use of “ value divided by cost ” in knapsack problems ( Badanidiyuru et al. , 2013 ) . In the knapsack problem , one selects items to include into a knapsack to maximize the sum of the selected items ’ values subject to a budget constraint on the sum of the items ’ costs . In this setting , myopically adding items to the knapsack that maximize value divided by cost has strong theoretical guarantees : this algorithm provides at least 1/2 the optimal value ( Williamson and Shmoys , 2011 ) . However , this theoretical guarantee relies on the additive nature of value in the knapsack problem . In contrast , in BO , the value obtained from multiple evaluations is the maximum of the values of the evaluations , not their sum . Indeed , we show that in this setting the “ value divided by cost ” approach can perform arbitrarily worse than the optimal policy . Heterogeneous evaluation costs have also been considered in the multi-armed bandits literature ( Badanidiyuru et al. , 2013 ; Xia et al. , 2015 , 2016 ) . These works develop algorithms based on optimistic policies that maximize some form of reward-to-cost ratio . As mentioned above , this type of policy is sensible when the measure of performance is the cumulative regret but is not appropriate in an optimization or “ best-arm identification ” setting . Our work is also closely related to non-myopic BO ( Gonzalez et al. , 2016 ; Lam et al. , 2016 ; Yue and Kontar , 2020 ; Jiang et al. , 2020a ; Lee et al. , 2020a ; Jiang et al. , 2020b ; Lee et al. , 2021 ) , a class of acquisition functions that account for future evaluations when quantifying a point ’ s acquisition value . To the best of our knowledge , the work of Lee et al . ( 2021 ) discussed above is the only one among these that is able to handle heterogeneous evaluation costs . 3 Budgeted Bayesian Optimization with Unknown Evaluation Costs . We now formally state the problem of budgeted BO with unknown evaluation costs . Given a compact optimization domain X ⊂ Rd , our goal is to find a point x ∈ X with the largest possible objective value by querying the black-box objective function , f : X → R , at a sequence of points { xi } ni=1 , subject to the constraint ∑n i=1 c ( xi ) ≤ B , where c ( x ) is the cost of evaluating f at x and B is the evaluation budget . The cost observation c ( x ) is revealed immediately after the evaluation of f ( x ) is performed . However , the actual cost function c is unknown . As is typical in BO , we endow f and c with a joint prior distribution , p. An observation in our setting is a triple ( xi , yi , zi ) ∈ X× R× R > 0 , 1This expression is for the case when c ( x ) is known . When c ( x ) is unknown and learned , then either the expectation is taken over the distribution of improvement to cost ratios ( as we do in our experiments ) or the denominator is replaced by the mean cost . where yi is an observation of the objective f at xi , and zi is an observation of the cost c for evaluating f at xi .

Summary: This paper considers the problem of limited budget Bayesian optimization when the cost of each experiment is unknown and must be estimated by running the experiment. This is an interesting problem introduced in the BO community recently. This paper improves upon previous approaches by treating the problem as a constrained MDP. At each step, the method performs a look-ahead and chooses the next point that maximizes a N-step reward within a budget constraint. The experiments show improved performance over the baselines.