Optimizing Conditional Value-At-Risk of Black-Box Functions

1 Introduction . A wide range of applications from Auto-ML [ 15 ] to chemistry [ 6 ] and drug design [ 3 ] require optimizing a black-box objective function ( i.e. , its closed-form expression , gradient , and convexity are unknown ) through observing noisy function evaluations . To resolve this problem , an efficient class of algorithms , called Bayesian optimization ( BO ) [ 2 , 20 ] , has seen rapid development lately . One of the recent developments of BO considers searching for the optimization variable x∗ that maximizes the objective function ρ [ f ( x , W ) ] [ 4 ] . Different from the classical BO , there exists an environmental random variable W that can not be controlled , which happens frequently in real-world problems . For example , to maximize the crop yield ( i.e. , f ( x , W ) ) by controlling the amount of fertilizer ( i.e. , x ) , there are various uncontrollable weather conditions such as the temperature , lighting , and rainfall ( i.e. , W ) [ 13 ] . Thus , even though an amount of fertilizer x increases the crop yield most of the time , there exists a risk/chance that under certain weather conditions ( realizations of W ) , the same amount of fertilizer leads to a low crop yield ( an undesirable realization of the random variable f ( x , W ) ) . This issue renders the optimization of f ( x , W ) futile , so the work of [ 4 ] proposes to optimize a risk measure , denoted as ρ , of f ( x , W ) such as value-at-risk ( VaR ) and conditional VaR ( CVaR ) . However , its approach suffers from two main shortcomings : the computational cost of a nested optimization procedure and the lack of theoretical performance guarantee . Although these drawbacks are addressed in the work of [ 13 ] , its proposed algorithm only works for optimizing VaR of a black-box function . Hence , an efficient algorithm with theoretical performance guarantee for optimizing CVaR of a black-box function remains an open question . Nonetheless , optimizing CVaR of a black-box function is desirable , especially when the effect of optimizing CVaR can not be obtained by optimizing VaR . A prominent property of CVaR is its sensitivity to values at the extreme tails of the random variable f ( x , W ) , which VaR does not exhibit [ 19 ] . Let us consider two portfolio allocation schemes : one scheme has an unacceptable 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . worst-case value of the return and the other does not . It is possible that the two schemes have the same VaR of the return because VaR is insensitive to the extreme tails of the return ( e.g. , the worst-case value of the return ) . On the other hand , CVaR is able to distinguish the risks of the two schemes . This advantage of CVaR is further clarified in Remark 1 . In this paper , we adopt the settings of [ 4 , 13 ] : the distribution of W is given and we can control/select the realization w of W in the optimization procedure . In practice , the distribution of W can be estimated from data ( e.g. , weather historical data in the crop yield optimization example ) and we can perform BO in a laboratory or using computer simulation where W is controlled [ 4 , 13 ] . After the optimization , the optimal x∗ can be used in the real-world environment with the uncontrollable W. Our main contribution is to propose two computationally efficient algorithms ( Sec . 3 ) with theoretical performance guarantee ( Sec . 4 ) for optimizing CVaR of a black-box function . First , we propose CVaR optimization with upper confidence bound ( CV-UCB ) that is based on the well-established principle of optimism in the face of uncertainty . By exploiting the connection between CVaR and VaR , we are able to develop a confidence bound of CVaR ( Sec . 3.1.1 ) and prove the no-regret guarantee of CV-UCB ( Sec . 4.1 ) . Second , we propose CVaR optimization with Thompson sampling ( CV-TS ) which can be naturally extended to handle batch queries ( i.e. , gathering observations at a batch of inputs in each BO iteration ) . The capability of gathering observations in a batch simultaneously is often available and preferable in laboratory settings and computer simulations . Though CV-TS can be simply performed with the popular Thompson sampling ( or posterior sampling ) [ 18 ] ( Sec . 3.1.2 ) , bounding its Bayesian regret is challenging as the distribution of CVaR in our problem has not been investigated before and its support is unbounded . Fortunately , we are able to relate the problem of bounding the tail expectation of CVaR to that of the function evaluation f ( x , w ) ( Sec . 4.2 ) . The latter follows a Gaussian distribution with known tail expectation , unlike that of CVaR . We empirically evaluate the performance of both CV-UCB and CV-TS in optimizing CVaR of synthetic benchmark functions , an optimization problem of the residuary resistance per unit weight of displacement of a yacht , a portfolio optimization problem , and a simulated robot task ( Sec . 5 ) . Related works . Existing solutions to optimizing a black-box function f ( x , W ) with an environmental random variable W are different in the assumptions about W and the objective function . Other than the aforementioned works of [ 4 , 13 ] where we adopt the assumptions that the distribution of W is known and w can be selected during the optimization procedure , the work of [ 9 ] also proposes an efficient algorithm with theoretical performance guarantee under the same assumptions to maximize the probabilistic threshold robustness measure . This measure requires a threshold of the desirable performance , instead of a risk level ( i.e. , the probability of undesirable performance ) in CVaR and VaR . Another related work is adversarially robust BO [ 1 ] which maximizes f ( x , w ) with the worst-case realization w of W. It can be cast as an optimization problem of VaR of a black-box function [ 13 ] . The work of [ 10 ] assumes that the distribution of W is given , but W is sampled from its distribution during the optimization . Its objective is to balance between the mean and the variance of f ( x , W ) from different perspectives : multi-task learning , multi-objective optimization , and constrained optimization . The work of [ 22 ] maximizes VaR and CVaR of a black-box function , yet , it assumes W is unobservable and its distribution is unknown . It suffers from an approximate posterior belief and the lack of theoretical performance guarantee . The works of [ 12 , 14 ] ( namely , distributional robust BO ) assume that the unknown distribution of W belongs to an uncertainty set of distributions . They maximize the mean of f ( x , W ) under the worst-case realization of the unknown distribution of W in the uncertainty set . 2 Background . 2.1 Conditional Value-at-Risk . Let the black-box function of interest be a real-valued function f ( x , w ) where x ∈ X ⊂ Rm and w ∈W ⊂ Rn . Let W denote the environmental random variable whose probability distribution is specified by p ( W = w ) for all w ∈W . Let f ( x , W ) denote the random variable representing the function evaluation at x and a random realization of W. The conditional value-at-risk ( CVaR ) of f ( x , W ) at risk level α ∈ ( 0 , 1 ) is defined as : cf ( x ; α ) , E [ f ( x , W ) | f ( x , W ) ≤ vf ( x ; α ) ] ( 1 ) which is the expectation of f ( x , W ) over function values that are at most the value-at-risk ( VaR ) at the same risk level α defined as vf ( x ; α ) , inf { ω : P ( f ( x , W ) ≤ ω ) ≥ α } . Remark 1 ( CVaR vs. VaR ) . While VaR at α is interpreted as the α-percentile of f ( x , W ) , CVaR is interpreted as the expectation of function values that are at most VaR . Therefore , CVaR is sensitive to the function values less than VaR , while VaR is not . Fig . 1 shows an example of function evaluations at x0 and x1 such that their VaR values ( i.e. , their α-percentiles ) are the same . We observe that VaR is insensitive to the tails of f ( x , W ) which are different in this case ( see the region around the black dot ) . In contrast , cf ( x0 ; α ) > cf ( x1 ; α ) which conveys the idea that f ( x0 , W ) has a lower risk of low function values as the probability density of function values less than VaR concentrates in a region nearer to VaR than that of f ( x1 , W ) . Thus , CVaR is more risk-aversed than VaR , especially when the distribution of f ( x , W ) has a heavy lower tail , which makes it more suitable for critical applications . In the portfolio optimization problem , if the distribution of the portfolio ’ s return has negative outliers ( which are captured in CVaR ) , the average return in the long run can be negative even though the VaR of the return is positive . In this work , we assume that |W| is finite to simplify the theoretical analysis ( described in Sec . 4 ) and the evaluation of CVaR . The latter requires the computation of the cumulative probability mass of f ( x , W ) by enumerating all possible realizations of f ( x , W ) ( described in [ 4 ] ) . 2.2 Bayesian Optimization of CVaR and Gaussian Processes . BO of CVaR . In this paper , we would like to maximize CVaR cf ( x ; α ) ( 1 ) of f ( x , W ) given the distribution p ( W ) by iteratively gathering observations as noisy function evaluations y ( x , w ) , f ( x , w ) + ( x , w ) where ( x , w ) ∼ N ( 0 , σ2n ) . The gist of BO of CVaR is a strategy to select the input query ( xt , wt ) at iteration t such that the maximizer x∗ ∈ argmaxx∈X cf ( x ; α ) is found as rapidly as possible . Note that while W is controllable in BO , after the optimization , the optimal x∗ can be used in real-world settings where W is uncontrollable . At iteration t , we have all observations at input queries in the previous t − 1 iterations , denote as Dt−1 = Dt−2 ∪ { ( xt−1 , wt−1 ) } ( D0 denotes the set of initial observed inputs ) . These observations yDt−1 , ( y ( x , w ) ) ( x , w ) ∈Dt−1 are used to construct the posterior belief/distribution of the underlying black-box function f ( x , w ) which is utilized in building the query selection strategy . Let us revisit Fig . 1 : cf ( x0 ; α ) > cf ( x1 ; α ) implies that x0 is preferred to x1 . It is because the risk/chance of getting low function values at x0 is smaller ( see the region around the black dot ) . In contrast , VaR can not differentiate the risk of f ( x0 , W ) and that of f ( x1 , W ) since their VaR values are the same . Algorithm 1 BO Algorithms for optimizing CVaR of a black-box function 1 : Input : algo , X , W , initial observation yD0 , prior µ0 = 0 , σn , κ 2 : for t = 1 , 2 , . . . do 3 : { Selecting xt } 4 : if algo is CV-UCB then 5 : Select xt ∈ argmax x cut−1 ( x ; α ) 6 : else if algo is CV-TS then 7 : Sample a function f1 from the GP posterior belief given yDt−1 8 : Select xt ∈ argmax x cf1 ( x ; α ) 9 : end if 10 : { Selecting wt } 11 : Find αt ∈ arg max α′∈ ( 0 , α ] vut−1 ( xt ; α ′ ) − vlt−1 ( xt ; α′ ) 12 : Given αt , select wt as an LV w.r.t . xt , ut−1 , and lt−1 13 : { Collecting data and updating GP } 14 : Incorporate new observation at input query : yDt = yDt−1 ∪ { y ( xt , wt ) } 15 : Update the GP posterior belief given yDt 16 : end for Gaussian process ( GP ) . In order to obtain the posterior belief of f ( x , w ) given noisy observations yDt−1 , we model the black-box function f with a GP : every finite subset of { f ( x , w ) } ( x , w ) ∈X×W follows a multivariate Gaussian distribution [ 17 ] . Thus , GP is fully specified by its prior mean which is assumed to be zero and its covariance function/kernel κ ( x , w ; x′ , w′ ) , cov [ f ( x , w ) , f ( x′ , w′ ) ] . Given noisy observations yDt−1 at iteration t , the posterior belief p ( f ( x , w ) |yDt−1 ) is a Gaussian specified by the following mean and variance : µt−1 ( x , w ) , Kx , w ; Dt−1ΛDt−1yDt−1 σ2t−1 ( x , w ) , κ ( x , w ) −Kx , w ; Dt−1ΛDt−1KDt−1 ; x , w ( 2 ) where κ ( x , w ) , κ ( x , w ; x , w ) , Kx , w ; Dt−1 , ( κ ( x , w ; x ′ , w′ ) ) ( x′ , w′ ) ∈Dt−1 , KDt−1 ; x , w , K > x , w ; Dt−1 , and ΛDt−1 , ( KDt−1Dt−1 + σ 2 nI ) −1 ( I is the identity matrix and KDt−1Dt−1 , ( κ ( x′ , w′ ; x′′ , w′′ ) ) ( x′ , w′ , x′′ , w′′ ) ∈Dt−1×Dt−1 ) . Although f ( x , w ) follows a Gaussian distribution , CVaR cf ( x ; α ) ( defined in ( 1 ) ) does not . Furthermore , although |W| is finite , the distribution of CVaR is continuous because the black-box function evaluation is real-valued . Besides , as the support of f ( x , W ) is unbounded , so is CVaR . These points pose challenges in deriving the following 2 algorithms and their theoretical analyses .

Summary: This paper considers a robust/risk-averse Bayesian optimization problem. In particular, the authors focus on maximizing conditional value at risk (CVaR), when the objective is unknown.  They propose two algorithms based on UCB and Thompson sampling strategies. Both algorithms are tested on synthetic datasets/tasks.