Risk-averse Heteroscedastic Bayesian Optimization

Many black-box optimization tasks arising in high-stakes applications require riskaverse decisions . The standard Bayesian optimization ( BO ) paradigm , however , optimizes the expected value only . We generalize BO to trade mean and inputdependent variance of the objective , both of which we assume to be unknown a priori . In particular , we propose a novel risk-averse heteroscedastic Bayesian optimization algorithm ( RAHBO ) that aims to identify a solution with high return and low noise variance , while learning the noise distribution on the fly . To this end , we model both expectation and variance as ( unknown ) RKHS functions , and propose a novel risk-aware acquisition function . We bound the regret for our approach and provide a robust rule to report the final decision point for applications where only a single solution must be identified . We demonstrate the effectiveness of RAHBO on synthetic benchmark functions and hyperparameter tuning tasks . 1 Introduction . Black-box optimization tasks arise frequently in high-stakes applications such as drug and material discovery [ 17 , 22 , 28 ] , genetics [ 16 , 27 ] , robotics [ 5 , 12 , 25 ] , hyperparameter tuning of complex learning systems [ 13 , 21 , 34 ] , to name a few . In many of these applications , there is often a trade-off between achieving high utility and minimizing risk . Moreover , uncertain and costly evaluations are an inherent part of black-box optimization tasks , and modern learning methods need to handle these aspects when balancing between the previous two objectives . Bayesian optimization ( BO ) is a powerful framework for optimizing such costly black-box functions from noisy zeroth-order evaluations . Classical BO approaches are typically risk-neutral as they seek to optimize the expected function value only . In practice , however , two different solutions might attain similar expected function values , but one might produce significantly noisier realizations . This is of major importance when it comes to actual deployment of the found solutions . For example , when selecting hyperparameters of a machine learning algorithm , we might prefer configurations that lead to slightly higher test errors but at the same time lead to smaller variance . In this paper , we generalize BO to trade off mean and input-dependent noise variance when sequentially querying points and outputting final solutions . We introduce a practical setting where both the black-box objective and input-dependent noise variance are unknown a priori , and the learner needs to estimate them on the fly . We propose a novel optimistic risk-averse algorithm – RAHBO – that makes sequential decisions by simultaneously balancing between exploration ( learning about uncertain actions ) , exploitation ( choosing actions that lead to high gains ) and risk ( avoiding unreliable actions ) . We bound the cumulative regret of RAHBO as well as the number of samples required to output a single near-optimal risk-averse solution . In our experiments , we demonstrate the risk-averse performance of our algorithm and show that standard BO methods can severely fail in applications where reliability of the reported solutions is of utmost importance . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Related work . Bayesian optimization ( BO ) [ 26 ] refers to approaches for optimizing a noisy blackbox objective that is typically expensive to evaluate . A great number of BO methods have been developed over the years , including a significant number of variants of popular algorithms such as GP-UCB [ 35 ] , Expected Improvement [ 26 ] , and Thompson Sampling [ 14 ] . While the focus of standard BO approaches is mainly on trading-off exploration vs. exploitation and optimizing for the expected performance , in this work , we additionally focus on the risk that is involved when working with noisy objectives , as illustrated in Figure 1 . The vast majority of previous BO works assume ( sub- ) Gaussian and homoscedastic noise ( i.e. , input independent and of some known fixed level ) . Both assumptions can be restrictive in practice . For example , as demonstrated in [ 15 ] , the majority of hyperparameter tuning tasks exhibit heteroscedasticity . A few works relax the first assumption and consider , e.g. , heavy-tailed noise models [ 31 ] and adversarially corrupted observations [ 8 ] . The second assumption is typically generalized via heteroscedastic Gaussian process ( GP ) , allowing an explicit dependence of the noise distribution on the evaluation points [ 6 , 7 , 10 , 20 ] . Similarly , in this work , we consider heteroscedastic GP models , but unlike the previous works , we specifically focus on the risk that is associated with querying and reporting noisy points . Several works have recently considered robust and risk-averse aspects in BO . Their central focus is on designing robust strategies and protecting against the change/shift in uncontrollable covariates . They study various notions including worst-case robustness [ 9 ] , distributional robustness [ 19 , 30 ] , robust mixed strategies [ 33 ] and other notions of risk-aversion [ 11 , 18 , 29 ] , and while some of them report robust regret guarantees , their focus is primarily on the robustness in the homoscedastic GP setting . Instead , in our setting , we account for the risk that comes from the realization of random noise with unknown distribution . Rather than optimizing the expected performance , in our risk-averse setting , we prefer inputs with lower variance . To this end , we incorporate the learning of the noise distribution into the optimization procedure via a mean-variance objective . The closest to our setting is risk-aversion with respect to noise in multi-armed bandits [ 32 ] . Their approach , however , fails to exploit correlation in rewards among similar arms . Contributions . We propose a novel Risk-averse Heteroscedastic Bayesian optimization ( RAHBO ) approach based on the optimistic principle that trades off the expectation and uncertainty of the mean-variance objective function . We model both the objective and variance as ( unknown ) functions belonging to RKHS space of functions , and propose a practical risk-aware algorithm in the heteroscedastic GP setting . In our theoretical analysis , we establish rigorous sublinear regret guarantees for our algorithm , and provide a robust reporting rule for applications where only a single solution is required . We demonstrate the effectiveness of RAHBO on synthetic benchmarks , as well as on hyperparameter tuning tasks for the Swiss free-electron laser and a machine learning model . 2 Problem setting . Let X be a given compact set of inputs ( X ⊂ Rd for some d ∈ N ) . We consider a problem of sequentially interacting with a fixed and unknown objective f : X → R. At every round of this procedure , the learner selects an action xt ∈ X , and obtains a noisy observation yt = f ( xt ) + ξ ( xt ) , ( 1 ) where ξ ( xt ) is zero-mean noise independent across different time steps t. In this work , we consider sub-Gaussian heteroscedastic noise that depends on the query location . Definition 1 . A zero-mean real-valued random variable ξ is ρ–sub-Gaussian , if there exists varianceproxy ρ2 such that ∀λ ∈ R , E [ eλξ ] ≤ e λ2ρ2 2 . For a sub-Gaussian ξ , its variance Var [ ξ ] lower bounds any valid variance-proxy ρ , i.e. , Var [ ξ ] ≤ ρ2 . In case Var [ ξ ] = ρ2 holds , ξ is said to be strictly ρ–sub-Gaussian . Besides zero-mean Gaussian random variables , most standard symmetric bounded random variables ( e.g. , Bernoulli , beta , uniform , binomial ) are strictly sub-Gaussian ( see [ 2 , Proposition 1.1 ] ) . Throughout the paper , we consider sub-Gaussian noise , and in Section 3.3 , we specialize to the case of strictly sub-Gaussian noise . Optimization objective . Unlike the previous works that mostly focus on sequential optimization of f in the homoscedastic noise case , in this work , we consider the trade-off between risk and return in the heteroscedastic case . While there exist a number of risk-averse objectives , we consider the simple and frequently used mean-variance objective ( MV ) [ 32 ] . Here , the objective value at x ∈ X is a trade-off between the ( mean ) return f ( x ) and the risk expressed by its variance-proxy ρ2 ( x ) : MV ( x ) = f ( x ) − αρ2 ( x ) , ( 2 ) where α ≥ 0 is a so-called coefficient of absolute risk tolerance . In this work , we assume α is fixed and known to the learner . In the case of α = 0 , maximizing MV ( x ) coincides with the standard BO objective . Performance metrics . We aim to construct a sequence of input evaluations xt that eventually maximizes the risk-averse objective MV ( x ) . To assess this convergence , we consider two metrics . The first metric corresponds to the notion of cumulative regret similar to the one used in standard BO and bandits . Here , the learner ’ s goal is to maximize its risk-averse cumulative reward over a time horizon T , or equivalently minimize its risk-averse cumulative regret : RT = T∑ t=1 [ MV ( x∗ ) −MV ( xt ) ] , ( 3 ) where x∗ ∈ arg maxx∈X MV ( x ) . A sublinear growth of RT with T implies vanishing average regret RT /T → 0 as T →∞ . Intuitively , this implies the existence of some t such that MV ( xt ) is arbitrarily close to the optimal value MV ( x∗ ) . The second metric is used when the learner seeks to simultaneously minimize the number of expensive function evaluations T . Namely , for a given accuracy ≥ 0 , we report a single `` good '' risk-averse point x̂T ∈ X after a total of T rounds , that satisfies : MV ( x̂T ) ≥ MV ( x∗ ) − . ( 4 ) Both metrics are important for choosing risk-averse solutions and which one is preferred depends on the application at hand . For example , risk-averse cumulative regret RT might be of a greater interest in online recommendation systems , while reporting a single point with high MV value might be more suitable when tuning machine learning hyperparameters . We consider both performance metrics in our experiments . Regularity assumptions . We consider standard smoothness assumptions [ 9 , 35 ] when it comes to the unknown function f : X → R. In particular , we assume that f ( · ) belongs to a reproducing kernel Hilbert space ( RKHS ) Hκ ( a space of smooth and real-valued functions defined on X ) , i.e. , f ∈ Hκ , induced by a kernel function κ ( · , · ) . We also assume that κ ( x , x′ ) ≤ 1 for every x , x′ ∈ X . Moreover , the RKHS norm of f ( · ) is assumed to be bounded ‖f‖κ ≤ Bf for some fixed constant Bf > 0 . We assume that the noise ξ ( x ) is ρ ( x ) –sub-Gaussian with variance-proxy ρ2 ( x ) uniformly bounded ρ ( x ) ∈ [ % , % ̄ ] for some constant values % ̄ ≥ % > 0 . 3 Algorithms . We first recall the Gaussian process ( GP ) based framework for sequential learning of RKHS functions from observations with heteroscedastic noise . Then , in Section 3.2 , we consider a simple risk-averse Bayesian optimization problem with known variance-proxy , and later on in Section 3.3 , we focus on our main problem setting in which the variance-proxy is unknown .

Summary: This paper proposes a risk-averse Bayesian optimization algorithm, which aims to find an input with both large expected return and small noise-dependent noise variance. Both the mean function and input-dependent noise variance are assumed to be unknown functions from a reproducing kernel Hilbert space, and hence both need to be estimated online. Both theoretical guarantees and empirical evaluations are given.