Risk-averse Heteroscedastic Bayesian Optimization

Many black-box optimization tasks arising in high-stakes applications require riskaverse decisions . The standard Bayesian optimization ( BO ) paradigm , however , optimizes the expected value only . We generalize BO to trade mean and inputdependent variance of the objective , both of which we assume to be unknown a priori . In particular , we propose a novel risk-averse heteroscedastic Bayesian optimization algorithm ( RAHBO ) that aims to identify a solution with high return and low noise variance , while learning the noise distribution on the fly . To this end , we model both expectation and variance as ( unknown ) RKHS functions , and propose a novel risk-aware acquisition function . We bound the regret for our approach and provide a robust rule to report the final decision point for applications where only a single solution must be identified . We demonstrate the effectiveness of RAHBO on synthetic benchmark functions and hyperparameter tuning tasks . 1 Introduction . Black-box optimization tasks arise frequently in high-stakes applications such as drug and material discovery [ 17 , 22 , 28 ] , genetics [ 16 , 27 ] , robotics [ 5 , 12 , 25 ] , hyperparameter tuning of complex learning systems [ 13 , 21 , 34 ] , to name a few . In many of these applications , there is often a trade-off between achieving high utility and minimizing risk . Moreover , uncertain and costly evaluations are an inherent part of black-box optimization tasks , and modern learning methods need to handle these aspects when balancing between the previous two objectives . Bayesian optimization ( BO ) is a powerful framework for optimizing such costly black-box functions from noisy zeroth-order evaluations . Classical BO approaches are typically risk-neutral as they seek to optimize the expected function value only . In practice , however , two different solutions might attain similar expected function values , but one might produce significantly noisier realizations . This is of major importance when it comes to actual deployment of the found solutions . For example , when selecting hyperparameters of a machine learning algorithm , we might prefer configurations that lead to slightly higher test errors but at the same time lead to smaller variance . In this paper , we generalize BO to trade off mean and input-dependent noise variance when sequentially querying points and outputting final solutions . We introduce a practical setting where both the black-box objective and input-dependent noise variance are unknown a priori , and the learner needs to estimate them on the fly . We propose a novel optimistic risk-averse algorithm – RAHBO – that makes sequential decisions by simultaneously balancing between exploration ( learning about uncertain actions ) , exploitation ( choosing actions that lead to high gains ) and risk ( avoiding unreliable actions ) . We bound the cumulative regret of RAHBO as well as the number of samples required to output a single near-optimal risk-averse solution . In our experiments , we demonstrate the risk-averse performance of our algorithm and show that standard BO methods can severely fail in applications where reliability of the reported solutions is of utmost importance . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Related work . Bayesian optimization ( BO ) [ 26 ] refers to approaches for optimizing a noisy blackbox objective that is typically expensive to evaluate . A great number of BO methods have been developed over the years , including a significant number of variants of popular algorithms such as GP-UCB [ 35 ] , Expected Improvement [ 26 ] , and Thompson Sampling [ 14 ] . While the focus of standard BO approaches is mainly on trading-off exploration vs. exploitation and optimizing for the expected performance , in this work , we additionally focus on the risk that is involved when working with noisy objectives , as illustrated in Figure 1 . The vast majority of previous BO works assume ( sub- ) Gaussian and homoscedastic noise ( i.e. , input independent and of some known fixed level ) . Both assumptions can be restrictive in practice . For example , as demonstrated in [ 15 ] , the majority of hyperparameter tuning tasks exhibit heteroscedasticity . A few works relax the first assumption and consider , e.g. , heavy-tailed noise models [ 31 ] and adversarially corrupted observations [ 8 ] . The second assumption is typically generalized via heteroscedastic Gaussian process ( GP ) , allowing an explicit dependence of the noise distribution on the evaluation points [ 6 , 7 , 10 , 20 ] . Similarly , in this work , we consider heteroscedastic GP models , but unlike the previous works , we specifically focus on the risk that is associated with querying and reporting noisy points . Several works have recently considered robust and risk-averse aspects in BO . Their central focus is on designing robust strategies and protecting against the change/shift in uncontrollable covariates . They study various notions including worst-case robustness [ 9 ] , distributional robustness [ 19 , 30 ] , robust mixed strategies [ 33 ] and other notions of risk-aversion [ 11 , 18 , 29 ] , and while some of them report robust regret guarantees , their focus is primarily on the robustness in the homoscedastic GP setting . Instead , in our setting , we account for the risk that comes from the realization of random noise with unknown distribution . Rather than optimizing the expected performance , in our risk-averse setting , we prefer inputs with lower variance . To this end , we incorporate the learning of the noise distribution into the optimization procedure via a mean-variance objective . The closest to our setting is risk-aversion with respect to noise in multi-armed bandits [ 32 ] . Their approach , however , fails to exploit correlation in rewards among similar arms . Contributions . We propose a novel Risk-averse Heteroscedastic Bayesian optimization ( RAHBO ) approach based on the optimistic principle that trades off the expectation and uncertainty of the mean-variance objective function . We model both the objective and variance as ( unknown ) functions belonging to RKHS space of functions , and propose a practical risk-aware algorithm in the heteroscedastic GP setting . In our theoretical analysis , we establish rigorous sublinear regret guarantees for our algorithm , and provide a robust reporting rule for applications where only a single solution is required . We demonstrate the effectiveness of RAHBO on synthetic benchmarks , as well as on hyperparameter tuning tasks for the Swiss free-electron laser and a machine learning model . 2 Problem setting . Let X be a given compact set of inputs ( X ⊂ Rd for some d ∈ N ) . We consider a problem of sequentially interacting with a fixed and unknown objective f : X → R. At every round of this procedure , the learner selects an action xt ∈ X , and obtains a noisy observation yt = f ( xt ) + ξ ( xt ) , ( 1 ) where ξ ( xt ) is zero-mean noise independent across different time steps t. In this work , we consider sub-Gaussian heteroscedastic noise that depends on the query location . Definition 1 . A zero-mean real-valued random variable ξ is ρ–sub-Gaussian , if there exists varianceproxy ρ2 such that ∀λ ∈ R , E [ eλξ ] ≤ e λ2ρ2 2 . For a sub-Gaussian ξ , its variance Var [ ξ ] lower bounds any valid variance-proxy ρ , i.e. , Var [ ξ ] ≤ ρ2 . In case Var [ ξ ] = ρ2 holds , ξ is said to be strictly ρ–sub-Gaussian . Besides zero-mean Gaussian random variables , most standard symmetric bounded random variables ( e.g. , Bernoulli , beta , uniform , binomial ) are strictly sub-Gaussian ( see [ 2 , Proposition 1.1 ] ) . Throughout the paper , we consider sub-Gaussian noise , and in Section 3.3 , we specialize to the case of strictly sub-Gaussian noise . Optimization objective . Unlike the previous works that mostly focus on sequential optimization of f in the homoscedastic noise case , in this work , we consider the trade-off between risk and return in the heteroscedastic case . While there exist a number of risk-averse objectives , we consider the simple and frequently used mean-variance objective ( MV ) [ 32 ] . Here , the objective value at x ∈ X is a trade-off between the ( mean ) return f ( x ) and the risk expressed by its variance-proxy ρ2 ( x ) : MV ( x ) = f ( x ) − αρ2 ( x ) , ( 2 ) where α ≥ 0 is a so-called coefficient of absolute risk tolerance . In this work , we assume α is fixed and known to the learner . In the case of α = 0 , maximizing MV ( x ) coincides with the standard BO objective . Performance metrics . We aim to construct a sequence of input evaluations xt that eventually maximizes the risk-averse objective MV ( x ) . To assess this convergence , we consider two metrics . The first metric corresponds to the notion of cumulative regret similar to the one used in standard BO and bandits . Here , the learner ’ s goal is to maximize its risk-averse cumulative reward over a time horizon T , or equivalently minimize its risk-averse cumulative regret : RT = T∑ t=1 [ MV ( x∗ ) −MV ( xt ) ] , ( 3 ) where x∗ ∈ arg maxx∈X MV ( x ) . A sublinear growth of RT with T implies vanishing average regret RT /T → 0 as T →∞ . Intuitively , this implies the existence of some t such that MV ( xt ) is arbitrarily close to the optimal value MV ( x∗ ) . The second metric is used when the learner seeks to simultaneously minimize the number of expensive function evaluations T . Namely , for a given accuracy ≥ 0 , we report a single `` good '' risk-averse point x̂T ∈ X after a total of T rounds , that satisfies : MV ( x̂T ) ≥ MV ( x∗ ) − . ( 4 ) Both metrics are important for choosing risk-averse solutions and which one is preferred depends on the application at hand . For example , risk-averse cumulative regret RT might be of a greater interest in online recommendation systems , while reporting a single point with high MV value might be more suitable when tuning machine learning hyperparameters . We consider both performance metrics in our experiments . Regularity assumptions . We consider standard smoothness assumptions [ 9 , 35 ] when it comes to the unknown function f : X → R. In particular , we assume that f ( · ) belongs to a reproducing kernel Hilbert space ( RKHS ) Hκ ( a space of smooth and real-valued functions defined on X ) , i.e. , f ∈ Hκ , induced by a kernel function κ ( · , · ) . We also assume that κ ( x , x′ ) ≤ 1 for every x , x′ ∈ X . Moreover , the RKHS norm of f ( · ) is assumed to be bounded ‖f‖κ ≤ Bf for some fixed constant Bf > 0 . We assume that the noise ξ ( x ) is ρ ( x ) –sub-Gaussian with variance-proxy ρ2 ( x ) uniformly bounded ρ ( x ) ∈ [ % , % ̄ ] for some constant values % ̄ ≥ % > 0 . 3 Algorithms . We first recall the Gaussian process ( GP ) based framework for sequential learning of RKHS functions from observations with heteroscedastic noise . Then , in Section 3.2 , we consider a simple risk-averse Bayesian optimization problem with known variance-proxy , and later on in Section 3.3 , we focus on our main problem setting in which the variance-proxy is unknown .

Summary: This paper proposes a new UCB strategy for Bayesian optimization that focuses on mean-variance optimization instead of expected value. The authors argue that this setting is relevant for heteroscedastic settings where one seeks high mean values with lower risk. Regret bounds are given to provide theoretical results about the convergence of the proposed algorithm for this setting. Empirical results on two synthetic problems and two "real-world" problems show that the proposed approach returns solutions with a higher mean and lower risk than standard risk-neutral GP-UCB.