Optimal Best-Arm Identification Methods for Tail-Risk Measures

1 Introduction . Tail risk is a common term used to quantify losses occurring due to rare events , and has been an important topic in finance , insurance and other safety critical uncertain environments . [ 44 ] first formalized the problem of identifying optimal investment in financial assets as a multi-criteria optimization problem of maximizing the average return , while minimizing the risk ( measured via variance ) . Since then , several other risk measures have been considered . Lately , risk-measures based on tails of the distribution , like the conditional value-at-risk ( CVaR ) and value-at-risk ( VaR ) , have gained popularity in financial regulations and risk management ( see , [ 48 , 47 ] ) , where the underlying probability distributions are mostly heavy tailed ( i.e . having infinite moment generating function for all θ > 0 ) . Informally , for a probability measure η , VaR at level π ∈ ( 0 , 1 ) is the πth quantile for η , i.e. , the outcome below which there is exactly π mass . CVaR at level π is the conditional expectation of η , conditioned on values beyond the VaR at level π . See Section 2 for precise definitions , and [ 50 , 46 ] for applications of these risk measures in finance and optimization . As opposed to VaR , CVaR is a coherent risk-measure , and is a preferable metric ( see , [ 5 ] for precise definition and properties of coherence ) . Outside finance , these tail-risk measures are being used to control risk in operations management , for example , in inventory management [ 4 ] , supply chain management [ 51 ] , etc . Recently , coherent risk measures , especially CVaR , have also been used in connection with fairness in machine learning [ 58 ] . The importance of these risk measures in the sequential decision making set-up has well been acknowledged ( see , [ 49 , 42 ] ) . Typically in the stochastic multi-armed bandit ( MAB ) literature , the quality of an arm is measured using its mean . Tight asymptotic and finite time guarantees exist for different MAB problems with performance measured by the mean ( see , [ 27 , 36 , 16 , 13 , 1 , 7 , 53 ] ) . Also , see [ 12 ] for a survey of the variants of stochastic MAB problems . However , maximizing the average reward is not always the primary desirable objective . In clinical trials , for example , the treatment that is good on average might result in adverse outcomes for some patients . In finance , one 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . is typically interested in balancing the mean return with the risk of extreme losses . Risk sensitivity has been well studied in the online learning setting , where in each round , the player sees reward from every arm ( see , [ 24 , 56 ] ) . However , there is very limited work which incorporates these risk-measures into the MAB framework . In this paper , we provide a systematic approach for identifying the distribution ( or arm ) from a given finite set of distributions ( or arms ) with minimum tail-risk ( as measured by CVaR or VaR , or by a conic combination of mean and CVaR , which we will henceforth refer to as the “ mean-CVaR ” objective ) . Adopting the best-arm identification ( BAI ) framework of the stochastic MAB problem , we consider algorithms that generate samples from the given arms , and are δ-correct , i.e. , identify the correct answer ( arm with minimum VaR , CVaR or mean-CVaR ) with probability at least 1− δ , for some pre-specified confidence level δ . While ensuring δ-correctness , the aim is to minimize the number of samples needed by the algorithm before its termination . This is the typical fixed-confidence setting of the BAI MAB problem ( see , [ 37 , 2 ] ) . Variants of this problem have been widely studied in the literature , where the best-arm is the one with maximum mean ( see , [ 43 , 25 , 6 , 14 , 26 , 33 , 27 , 34 ] ) . A relaxation of the pure exploration setting described above is the ( , δ ) -PAC setting , where the aim is to output an -optimal arm ( for an appropriate notion of -optimality ) , with probability at least 1− δ , while minimizing the number of samples generated . [ 59 , 18 , 32 ] consider the pure exploration problem of identifying the arm with minimum risk in the ( , δ ) -PAC setting . While [ 59 ] consider both VaR and CVaR as measures of risk , [ 18 , 32 ] focus on the VaR-problem . Recently , [ 39 ] and [ 35 ] have studied the BAI MAB problem with CVaR and mean-CVaR objectives , respectively , in the closely related “ fixed-budget ” framework , in which the total number of samples the algorithm is allowed to take is fixed , and the aim of the algorithm is to minimize the error-probability . 1.1 Outline of the approach and main assumption . As a warm-up , we first solve our minimum tail risk identification problems in the simple commonplace setting of arm-distributions belonging to a canonical single parameter exponential family ( SPEF ) of distributions . Each distribution in this family is uniquely identified with its parameter . We show that both CVaR and VaR are monotonic functions of this parameter , as is the mean . Hence , finding the best- ( CVaR/VaR/mean-CVaR ) arm reduces to finding the arm with the minimum mean . Since risk-sensitive objectives are particularly important when there is a non-trivial probability of occurrence of extreme outcomes , it is important to consider arm-distributions beyond canonical SPEF , for which the above-mentioned equivalence breaks . We solve the VaR problem for arbitrary arm distributions . In contrast the CVaR problem is unlearnable in full generality : on the class of all arm distributions , any δ-correct algorithm requires an infinite number of samples in expectation to identify the best arm amongst any finite collection of arms ( Remark 3.1 ) . To avoid this , we impose a mild and standard raw ( 1 + ) -moment restriction on the arm-distributions . Let P ( < ) denote the collection of all the probability distributions on the reals < , and let B and be positive constants . For risk measure CVaR and for the mean-CVaR objective , we restrict the class of allowed arm distributions to L = { η ∈ P ( < ) : Eη ( |X|1+ ) ≤ B } . We discuss the choice of parameters in Section 1.3 below . For each tail measure , we prove informationtheoretic lower bounds on the sample complexity of any δ-correct algorithm , and use these to develop a δ-correct algorithm whose sample complexity exactly matches the lower bound as δ → 0 , for CVaR , mean-CVaR , and VaR problems . The mean-CVaR problem is conceptually and technically similar to the CVaR problem . Hence , for simplicity of presentation , we primarily focus on the CVaR setting in the main text and give details of the mean-CVaR setting in Appendix I . We also spell out the somewhat analogous analysis for the VaR setting towards the end ( Section 4.2 ) , with details deferred Appendix to H . 1.2 Technical contributions . As is well known in the BAI MAB literature , the lower bound problem takes the underlying arm distributions as inputs and solves for optimal weights that determine the proportion of samples that should ideally be allocated to each arm . The proposed algorithm uses a plugin strategy that at each sequential stage , modulo mild forced exploration , uses the generated empirical distributions as a proxy for the true distributions and arrives at weights that guide the sequential sampling strategy . In order to highlight the technical challenges arising in our non-parametric case , we will need to introduce two functionals next that are central to our lower bounds , algorithms , confidence intervals etc . Information distance for CVaR problem : Given η1 , η2 in P ( < ) , let KL ( η1 , η2 ) denote the KLdivergence between them , i.e. , KL ( η1 , η2 ) : = ∫ log dη1dη2 ( y ) dη1 ( y ) . Furthermore , for the probability measure η let cπ ( η ) denote its CVaR at the given confidence level π ∈ ( 0 , 1 ) ( see Section 2 for the exact definition ) . Then , given η ∈ P ( < ) and x ∈ < , we define functionals KLUinf : P ( < ) × < −→ < + , and KLLinf : P ( < ) × < −→ < + , where < + denotes the non-negative reals , as KLUinf ( η , x ) : = min κ∈L : cπ ( κ ) ≥x KL ( η , κ ) and KLLinf ( η , x ) : = min κ∈L : cπ ( κ ) ≤x KL ( η , κ ) . ( 1 ) See [ 2 , 30 , 15 ] for related quantities . These projection functionals appear in the lower bound ( Section 3 ) , and are central to our plugin algorithm . Unlike their analogues in the mean case , KLUinf and KL L inf in ( 1 ) are not symmetric , and need to be studied separately . In particular , KLUinf is a convex optimization problem , while KL L inf is not . This is because cπ ( · ) is a concave function , whence , the CVaR constraint in the KLLinf problem in ( 1 ) renders the feasible region non-convex ( see Section 2 ) . CVaR can be expressed as the optimal value of a minimization problem . This helped in re-expressing KLLinf as minimization over 2 variables , fixing one of which resulted in convex optimization over the other ( see Section 3 ) . For proving δ-correctness , we develop a new concentration inequality for weighted sums of these functionals ( Proposition 4.2 ) . Dual representations of these suggest natural candidates for supermartingales , whose mixtures help us in proving the concentration result . Similar inequalities were developed in [ 38 , 20 , 54 ] in different settings . See [ 40 , Chapter 20 ] for an overview of the method of mixtures . We also propose KLUinf - and KL L inf -based tight anytime-valid confidence intervals for CVaR for heavy-tailed distributions , and show that classical confidence intervals derived using popular truncation-based estimators can be recovered using our method , with only a minor overhead ( see Section 4.3 ) . Since distributions in L are not characterized by parameters , we work in the space of probability measures instead of in the Euclidean space . A key and non-trivial requirement for the proof of asymptotic optimality of the algorithm is the joint continuity of KLLinf and KL U inf in a well-chosen metric , which should generate a topology that is sufficiently fine to ensure this continuity , but coarse to ensure fast convergence of the empirical distributions to the true-arm distributions . We endow P ( < ) with the topology of weak convergence , or equivalently , with the Lévy metric ( see Section 2 for definitions ) . Another nuance in our analysis is that the empirical distributions may not lie in L. This is handled by projecting these on to L under a suitable metric . Our proposed algorithm is a plugin strategy that involves solving the lower bound problem using the empirical distributions as a proxy for the actual arm distributions . This can be computationally demanding especially as the underlying samples in the empirical distribution become large . To ease the numerical burden we propose modifications that require solving the lower bound only order log ( n ) many times till stage n of the algorithm ( where n samples are generated ) . This modification substantially reduces the computation burden . We show that it is optimal up to a constant ( Appendix K ) . VaR problem : Our algorithm for CVaR , with KLLinf and KLUinf replaced by the corresponding functionals with the VaR constraints instead , is asymptotically optimal for this problem in complete generality ( Section 4.2 ) . Here , KLUinf and KL L inf have closed form representations . However , they are no longer jointly-continuous in the Lévy metric , which introduces new technical challenges in the analysis of the algorithm .

Summary: The authors study the problem of multi armed bandits with the goal of identifying the arm with smallest (i) CVaR (ii) VaR or (iii) conic combination of CVaR and mean. This problem has applications in finance and clinical trials and captures a notion of risk sensitivity towards undesirable outcomes. The authors show that in the parametric case where the arm reward distributions follow a canonical SPEF, the problem reduces to best arm identification.  The authors study the VaR problem without any assumptions on the distributions, and study the CVaR problem assuming that distributions satisfy bounded $(1+\epsilon)$ moments. The proposed algorithm projects the empirical distribution to the considered family of distributions and considers an exploratory transform that guides the arm sampling distribution. At each time the empirical reward distribution suggests an arm with empirical minimum CVaR - this is the null hypothesis which is tested against all alternatives using GLRT. The algorithm upon satisfying the termination condition outputs the arms with the empirical minimum CVaR.  The authors show that the proposed algorithm asymptotically achieves the optimal sample complexity as the error threshold $\delta \to 0$. The results are supported by numerical simulations demonstrating good performance on instances with Bernoulli arms. The performance of the algorithm hinges on the asymptotic convergence of the track-and-stop exploration protocol which is often fast even when the error parameter delta is moderate.