Optimizing Conditional Value-At-Risk of Black-Box Functions

1 Introduction . A wide range of applications from Auto-ML [ 15 ] to chemistry [ 6 ] and drug design [ 3 ] require optimizing a black-box objective function ( i.e. , its closed-form expression , gradient , and convexity are unknown ) through observing noisy function evaluations . To resolve this problem , an efficient class of algorithms , called Bayesian optimization ( BO ) [ 2 , 20 ] , has seen rapid development lately . One of the recent developments of BO considers searching for the optimization variable x∗ that maximizes the objective function ρ [ f ( x , W ) ] [ 4 ] . Different from the classical BO , there exists an environmental random variable W that can not be controlled , which happens frequently in real-world problems . For example , to maximize the crop yield ( i.e. , f ( x , W ) ) by controlling the amount of fertilizer ( i.e. , x ) , there are various uncontrollable weather conditions such as the temperature , lighting , and rainfall ( i.e. , W ) [ 13 ] . Thus , even though an amount of fertilizer x increases the crop yield most of the time , there exists a risk/chance that under certain weather conditions ( realizations of W ) , the same amount of fertilizer leads to a low crop yield ( an undesirable realization of the random variable f ( x , W ) ) . This issue renders the optimization of f ( x , W ) futile , so the work of [ 4 ] proposes to optimize a risk measure , denoted as ρ , of f ( x , W ) such as value-at-risk ( VaR ) and conditional VaR ( CVaR ) . However , its approach suffers from two main shortcomings : the computational cost of a nested optimization procedure and the lack of theoretical performance guarantee . Although these drawbacks are addressed in the work of [ 13 ] , its proposed algorithm only works for optimizing VaR of a black-box function . Hence , an efficient algorithm with theoretical performance guarantee for optimizing CVaR of a black-box function remains an open question . Nonetheless , optimizing CVaR of a black-box function is desirable , especially when the effect of optimizing CVaR can not be obtained by optimizing VaR . A prominent property of CVaR is its sensitivity to values at the extreme tails of the random variable f ( x , W ) , which VaR does not exhibit [ 19 ] . Let us consider two portfolio allocation schemes : one scheme has an unacceptable 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . worst-case value of the return and the other does not . It is possible that the two schemes have the same VaR of the return because VaR is insensitive to the extreme tails of the return ( e.g. , the worst-case value of the return ) . On the other hand , CVaR is able to distinguish the risks of the two schemes . This advantage of CVaR is further clarified in Remark 1 . In this paper , we adopt the settings of [ 4 , 13 ] : the distribution of W is given and we can control/select the realization w of W in the optimization procedure . In practice , the distribution of W can be estimated from data ( e.g. , weather historical data in the crop yield optimization example ) and we can perform BO in a laboratory or using computer simulation where W is controlled [ 4 , 13 ] . After the optimization , the optimal x∗ can be used in the real-world environment with the uncontrollable W. Our main contribution is to propose two computationally efficient algorithms ( Sec . 3 ) with theoretical performance guarantee ( Sec . 4 ) for optimizing CVaR of a black-box function . First , we propose CVaR optimization with upper confidence bound ( CV-UCB ) that is based on the well-established principle of optimism in the face of uncertainty . By exploiting the connection between CVaR and VaR , we are able to develop a confidence bound of CVaR ( Sec . 3.1.1 ) and prove the no-regret guarantee of CV-UCB ( Sec . 4.1 ) . Second , we propose CVaR optimization with Thompson sampling ( CV-TS ) which can be naturally extended to handle batch queries ( i.e. , gathering observations at a batch of inputs in each BO iteration ) . The capability of gathering observations in a batch simultaneously is often available and preferable in laboratory settings and computer simulations . Though CV-TS can be simply performed with the popular Thompson sampling ( or posterior sampling ) [ 18 ] ( Sec . 3.1.2 ) , bounding its Bayesian regret is challenging as the distribution of CVaR in our problem has not been investigated before and its support is unbounded . Fortunately , we are able to relate the problem of bounding the tail expectation of CVaR to that of the function evaluation f ( x , w ) ( Sec . 4.2 ) . The latter follows a Gaussian distribution with known tail expectation , unlike that of CVaR . We empirically evaluate the performance of both CV-UCB and CV-TS in optimizing CVaR of synthetic benchmark functions , an optimization problem of the residuary resistance per unit weight of displacement of a yacht , a portfolio optimization problem , and a simulated robot task ( Sec . 5 ) . Related works . Existing solutions to optimizing a black-box function f ( x , W ) with an environmental random variable W are different in the assumptions about W and the objective function . Other than the aforementioned works of [ 4 , 13 ] where we adopt the assumptions that the distribution of W is known and w can be selected during the optimization procedure , the work of [ 9 ] also proposes an efficient algorithm with theoretical performance guarantee under the same assumptions to maximize the probabilistic threshold robustness measure . This measure requires a threshold of the desirable performance , instead of a risk level ( i.e. , the probability of undesirable performance ) in CVaR and VaR . Another related work is adversarially robust BO [ 1 ] which maximizes f ( x , w ) with the worst-case realization w of W. It can be cast as an optimization problem of VaR of a black-box function [ 13 ] . The work of [ 10 ] assumes that the distribution of W is given , but W is sampled from its distribution during the optimization . Its objective is to balance between the mean and the variance of f ( x , W ) from different perspectives : multi-task learning , multi-objective optimization , and constrained optimization . The work of [ 22 ] maximizes VaR and CVaR of a black-box function , yet , it assumes W is unobservable and its distribution is unknown . It suffers from an approximate posterior belief and the lack of theoretical performance guarantee . The works of [ 12 , 14 ] ( namely , distributional robust BO ) assume that the unknown distribution of W belongs to an uncertainty set of distributions . They maximize the mean of f ( x , W ) under the worst-case realization of the unknown distribution of W in the uncertainty set . 2 Background . 2.1 Conditional Value-at-Risk . Let the black-box function of interest be a real-valued function f ( x , w ) where x ∈ X ⊂ Rm and w ∈W ⊂ Rn . Let W denote the environmental random variable whose probability distribution is specified by p ( W = w ) for all w ∈W . Let f ( x , W ) denote the random variable representing the function evaluation at x and a random realization of W. The conditional value-at-risk ( CVaR ) of f ( x , W ) at risk level α ∈ ( 0 , 1 ) is defined as : cf ( x ; α ) , E [ f ( x , W ) | f ( x , W ) ≤ vf ( x ; α ) ] ( 1 ) which is the expectation of f ( x , W ) over function values that are at most the value-at-risk ( VaR ) at the same risk level α defined as vf ( x ; α ) , inf { ω : P ( f ( x , W ) ≤ ω ) ≥ α } . Remark 1 ( CVaR vs. VaR ) . While VaR at α is interpreted as the α-percentile of f ( x , W ) , CVaR is interpreted as the expectation of function values that are at most VaR . Therefore , CVaR is sensitive to the function values less than VaR , while VaR is not . Fig . 1 shows an example of function evaluations at x0 and x1 such that their VaR values ( i.e. , their α-percentiles ) are the same . We observe that VaR is insensitive to the tails of f ( x , W ) which are different in this case ( see the region around the black dot ) . In contrast , cf ( x0 ; α ) > cf ( x1 ; α ) which conveys the idea that f ( x0 , W ) has a lower risk of low function values as the probability density of function values less than VaR concentrates in a region nearer to VaR than that of f ( x1 , W ) . Thus , CVaR is more risk-aversed than VaR , especially when the distribution of f ( x , W ) has a heavy lower tail , which makes it more suitable for critical applications . In the portfolio optimization problem , if the distribution of the portfolio ’ s return has negative outliers ( which are captured in CVaR ) , the average return in the long run can be negative even though the VaR of the return is positive . In this work , we assume that |W| is finite to simplify the theoretical analysis ( described in Sec . 4 ) and the evaluation of CVaR . The latter requires the computation of the cumulative probability mass of f ( x , W ) by enumerating all possible realizations of f ( x , W ) ( described in [ 4 ] ) . 2.2 Bayesian Optimization of CVaR and Gaussian Processes . BO of CVaR . In this paper , we would like to maximize CVaR cf ( x ; α ) ( 1 ) of f ( x , W ) given the distribution p ( W ) by iteratively gathering observations as noisy function evaluations y ( x , w ) , f ( x , w ) + ( x , w ) where ( x , w ) ∼ N ( 0 , σ2n ) . The gist of BO of CVaR is a strategy to select the input query ( xt , wt ) at iteration t such that the maximizer x∗ ∈ argmaxx∈X cf ( x ; α ) is found as rapidly as possible . Note that while W is controllable in BO , after the optimization , the optimal x∗ can be used in real-world settings where W is uncontrollable . At iteration t , we have all observations at input queries in the previous t − 1 iterations , denote as Dt−1 = Dt−2 ∪ { ( xt−1 , wt−1 ) } ( D0 denotes the set of initial observed inputs ) . These observations yDt−1 , ( y ( x , w ) ) ( x , w ) ∈Dt−1 are used to construct the posterior belief/distribution of the underlying black-box function f ( x , w ) which is utilized in building the query selection strategy . Let us revisit Fig . 1 : cf ( x0 ; α ) > cf ( x1 ; α ) implies that x0 is preferred to x1 . It is because the risk/chance of getting low function values at x0 is smaller ( see the region around the black dot ) . In contrast , VaR can not differentiate the risk of f ( x0 , W ) and that of f ( x1 , W ) since their VaR values are the same . Algorithm 1 BO Algorithms for optimizing CVaR of a black-box function 1 : Input : algo , X , W , initial observation yD0 , prior µ0 = 0 , σn , κ 2 : for t = 1 , 2 , . . . do 3 : { Selecting xt } 4 : if algo is CV-UCB then 5 : Select xt ∈ argmax x cut−1 ( x ; α ) 6 : else if algo is CV-TS then 7 : Sample a function f1 from the GP posterior belief given yDt−1 8 : Select xt ∈ argmax x cf1 ( x ; α ) 9 : end if 10 : { Selecting wt } 11 : Find αt ∈ arg max α′∈ ( 0 , α ] vut−1 ( xt ; α ′ ) − vlt−1 ( xt ; α′ ) 12 : Given αt , select wt as an LV w.r.t . xt , ut−1 , and lt−1 13 : { Collecting data and updating GP } 14 : Incorporate new observation at input query : yDt = yDt−1 ∪ { y ( xt , wt ) } 15 : Update the GP posterior belief given yDt 16 : end for Gaussian process ( GP ) . In order to obtain the posterior belief of f ( x , w ) given noisy observations yDt−1 , we model the black-box function f with a GP : every finite subset of { f ( x , w ) } ( x , w ) ∈X×W follows a multivariate Gaussian distribution [ 17 ] . Thus , GP is fully specified by its prior mean which is assumed to be zero and its covariance function/kernel κ ( x , w ; x′ , w′ ) , cov [ f ( x , w ) , f ( x′ , w′ ) ] . Given noisy observations yDt−1 at iteration t , the posterior belief p ( f ( x , w ) |yDt−1 ) is a Gaussian specified by the following mean and variance : µt−1 ( x , w ) , Kx , w ; Dt−1ΛDt−1yDt−1 σ2t−1 ( x , w ) , κ ( x , w ) −Kx , w ; Dt−1ΛDt−1KDt−1 ; x , w ( 2 ) where κ ( x , w ) , κ ( x , w ; x , w ) , Kx , w ; Dt−1 , ( κ ( x , w ; x ′ , w′ ) ) ( x′ , w′ ) ∈Dt−1 , KDt−1 ; x , w , K > x , w ; Dt−1 , and ΛDt−1 , ( KDt−1Dt−1 + σ 2 nI ) −1 ( I is the identity matrix and KDt−1Dt−1 , ( κ ( x′ , w′ ; x′′ , w′′ ) ) ( x′ , w′ , x′′ , w′′ ) ∈Dt−1×Dt−1 ) . Although f ( x , w ) follows a Gaussian distribution , CVaR cf ( x ; α ) ( defined in ( 1 ) ) does not . Furthermore , although |W| is finite , the distribution of CVaR is continuous because the black-box function evaluation is real-valued . Besides , as the support of f ( x , W ) is unbounded , so is CVaR . These points pose challenges in deriving the following 2 algorithms and their theoretical analyses .

Summary: This paper considers Bayesian optimization (BO) of objective functions of the form $c_f(x;\alpha) = \mathrm{CVaR}_\alpha(f(x, W))$, where $f$ is modeled as a Gaussian process (GP),, and $\mathrm{CVaR}_\alpha$ denotes the $\alpha$-level conditional value-at-risk, which is computed with respect to the randomness over $W$. This work focuses on the simulation optimization setting, where it is possible to evaluate $f(x,w)$ for each $(x,w)\in\mathbb{X}\times\mathbb{W}$, and develops two acquisition functions to select the pairs $(x,w)$ at which to evaluate $f$, CV-UCB and CV-TS, which extend the classical UCB and Thompson sampling acquisition functions, respectively. These algorithms select the $x$ component of the pair and are complemented with a common strategy to select $w$. Given $x_t$, the $x$ component of the pair to be evaluated at time $t$, this strategy first chooses $\alpha_t$ that maximizes the difference of the VaR of an upper confidence bound on $f(x_t, W)$, $u_t(x, W)$, and the VaR of a lower confidence bound on $f(x_t, W)$, $l_t(x, W)$. Finally, given $x_t$ and $\alpha_t$, this strategy chooses $w_t$ so that   $[\mathrm{VaR}_\alpha (l_t(x, W)),  \mathrm{VaR}_\alpha (u_t(x, W))] \subset [l_t(x_t, w_t), u_t(x_t, w_t)]$ (for $\alpha= \alpha_t$). A $w_t$ satisfying this condition is called a lacing value. The two resulting algorithms are shown to enjoy sublinear cumulative regrets when $\mathbb{X}$ and $\mathbb{W}$ are both finite. Finally, the empirical performance of these algorithms is evaluated in three synthetic experiments and two real-world experiments, showing favorable results.