Optimal Best-Arm Identification Methods for Tail-Risk Measures

1 Introduction . Tail risk is a common term used to quantify losses occurring due to rare events , and has been an important topic in finance , insurance and other safety critical uncertain environments . [ 44 ] first formalized the problem of identifying optimal investment in financial assets as a multi-criteria optimization problem of maximizing the average return , while minimizing the risk ( measured via variance ) . Since then , several other risk measures have been considered . Lately , risk-measures based on tails of the distribution , like the conditional value-at-risk ( CVaR ) and value-at-risk ( VaR ) , have gained popularity in financial regulations and risk management ( see , [ 48 , 47 ] ) , where the underlying probability distributions are mostly heavy tailed ( i.e . having infinite moment generating function for all θ > 0 ) . Informally , for a probability measure η , VaR at level π ∈ ( 0 , 1 ) is the πth quantile for η , i.e. , the outcome below which there is exactly π mass . CVaR at level π is the conditional expectation of η , conditioned on values beyond the VaR at level π . See Section 2 for precise definitions , and [ 50 , 46 ] for applications of these risk measures in finance and optimization . As opposed to VaR , CVaR is a coherent risk-measure , and is a preferable metric ( see , [ 5 ] for precise definition and properties of coherence ) . Outside finance , these tail-risk measures are being used to control risk in operations management , for example , in inventory management [ 4 ] , supply chain management [ 51 ] , etc . Recently , coherent risk measures , especially CVaR , have also been used in connection with fairness in machine learning [ 58 ] . The importance of these risk measures in the sequential decision making set-up has well been acknowledged ( see , [ 49 , 42 ] ) . Typically in the stochastic multi-armed bandit ( MAB ) literature , the quality of an arm is measured using its mean . Tight asymptotic and finite time guarantees exist for different MAB problems with performance measured by the mean ( see , [ 27 , 36 , 16 , 13 , 1 , 7 , 53 ] ) . Also , see [ 12 ] for a survey of the variants of stochastic MAB problems . However , maximizing the average reward is not always the primary desirable objective . In clinical trials , for example , the treatment that is good on average might result in adverse outcomes for some patients . In finance , one 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . is typically interested in balancing the mean return with the risk of extreme losses . Risk sensitivity has been well studied in the online learning setting , where in each round , the player sees reward from every arm ( see , [ 24 , 56 ] ) . However , there is very limited work which incorporates these risk-measures into the MAB framework . In this paper , we provide a systematic approach for identifying the distribution ( or arm ) from a given finite set of distributions ( or arms ) with minimum tail-risk ( as measured by CVaR or VaR , or by a conic combination of mean and CVaR , which we will henceforth refer to as the “ mean-CVaR ” objective ) . Adopting the best-arm identification ( BAI ) framework of the stochastic MAB problem , we consider algorithms that generate samples from the given arms , and are δ-correct , i.e. , identify the correct answer ( arm with minimum VaR , CVaR or mean-CVaR ) with probability at least 1− δ , for some pre-specified confidence level δ . While ensuring δ-correctness , the aim is to minimize the number of samples needed by the algorithm before its termination . This is the typical fixed-confidence setting of the BAI MAB problem ( see , [ 37 , 2 ] ) . Variants of this problem have been widely studied in the literature , where the best-arm is the one with maximum mean ( see , [ 43 , 25 , 6 , 14 , 26 , 33 , 27 , 34 ] ) . A relaxation of the pure exploration setting described above is the ( , δ ) -PAC setting , where the aim is to output an -optimal arm ( for an appropriate notion of -optimality ) , with probability at least 1− δ , while minimizing the number of samples generated . [ 59 , 18 , 32 ] consider the pure exploration problem of identifying the arm with minimum risk in the ( , δ ) -PAC setting . While [ 59 ] consider both VaR and CVaR as measures of risk , [ 18 , 32 ] focus on the VaR-problem . Recently , [ 39 ] and [ 35 ] have studied the BAI MAB problem with CVaR and mean-CVaR objectives , respectively , in the closely related “ fixed-budget ” framework , in which the total number of samples the algorithm is allowed to take is fixed , and the aim of the algorithm is to minimize the error-probability . 1.1 Outline of the approach and main assumption . As a warm-up , we first solve our minimum tail risk identification problems in the simple commonplace setting of arm-distributions belonging to a canonical single parameter exponential family ( SPEF ) of distributions . Each distribution in this family is uniquely identified with its parameter . We show that both CVaR and VaR are monotonic functions of this parameter , as is the mean . Hence , finding the best- ( CVaR/VaR/mean-CVaR ) arm reduces to finding the arm with the minimum mean . Since risk-sensitive objectives are particularly important when there is a non-trivial probability of occurrence of extreme outcomes , it is important to consider arm-distributions beyond canonical SPEF , for which the above-mentioned equivalence breaks . We solve the VaR problem for arbitrary arm distributions . In contrast the CVaR problem is unlearnable in full generality : on the class of all arm distributions , any δ-correct algorithm requires an infinite number of samples in expectation to identify the best arm amongst any finite collection of arms ( Remark 3.1 ) . To avoid this , we impose a mild and standard raw ( 1 + ) -moment restriction on the arm-distributions . Let P ( < ) denote the collection of all the probability distributions on the reals < , and let B and be positive constants . For risk measure CVaR and for the mean-CVaR objective , we restrict the class of allowed arm distributions to L = { η ∈ P ( < ) : Eη ( |X|1+ ) ≤ B } . We discuss the choice of parameters in Section 1.3 below . For each tail measure , we prove informationtheoretic lower bounds on the sample complexity of any δ-correct algorithm , and use these to develop a δ-correct algorithm whose sample complexity exactly matches the lower bound as δ → 0 , for CVaR , mean-CVaR , and VaR problems . The mean-CVaR problem is conceptually and technically similar to the CVaR problem . Hence , for simplicity of presentation , we primarily focus on the CVaR setting in the main text and give details of the mean-CVaR setting in Appendix I . We also spell out the somewhat analogous analysis for the VaR setting towards the end ( Section 4.2 ) , with details deferred Appendix to H . 1.2 Technical contributions . As is well known in the BAI MAB literature , the lower bound problem takes the underlying arm distributions as inputs and solves for optimal weights that determine the proportion of samples that should ideally be allocated to each arm . The proposed algorithm uses a plugin strategy that at each sequential stage , modulo mild forced exploration , uses the generated empirical distributions as a proxy for the true distributions and arrives at weights that guide the sequential sampling strategy . In order to highlight the technical challenges arising in our non-parametric case , we will need to introduce two functionals next that are central to our lower bounds , algorithms , confidence intervals etc . Information distance for CVaR problem : Given η1 , η2 in P ( < ) , let KL ( η1 , η2 ) denote the KLdivergence between them , i.e. , KL ( η1 , η2 ) : = ∫ log dη1dη2 ( y ) dη1 ( y ) . Furthermore , for the probability measure η let cπ ( η ) denote its CVaR at the given confidence level π ∈ ( 0 , 1 ) ( see Section 2 for the exact definition ) . Then , given η ∈ P ( < ) and x ∈ < , we define functionals KLUinf : P ( < ) × < −→ < + , and KLLinf : P ( < ) × < −→ < + , where < + denotes the non-negative reals , as KLUinf ( η , x ) : = min κ∈L : cπ ( κ ) ≥x KL ( η , κ ) and KLLinf ( η , x ) : = min κ∈L : cπ ( κ ) ≤x KL ( η , κ ) . ( 1 ) See [ 2 , 30 , 15 ] for related quantities . These projection functionals appear in the lower bound ( Section 3 ) , and are central to our plugin algorithm . Unlike their analogues in the mean case , KLUinf and KL L inf in ( 1 ) are not symmetric , and need to be studied separately . In particular , KLUinf is a convex optimization problem , while KL L inf is not . This is because cπ ( · ) is a concave function , whence , the CVaR constraint in the KLLinf problem in ( 1 ) renders the feasible region non-convex ( see Section 2 ) . CVaR can be expressed as the optimal value of a minimization problem . This helped in re-expressing KLLinf as minimization over 2 variables , fixing one of which resulted in convex optimization over the other ( see Section 3 ) . For proving δ-correctness , we develop a new concentration inequality for weighted sums of these functionals ( Proposition 4.2 ) . Dual representations of these suggest natural candidates for supermartingales , whose mixtures help us in proving the concentration result . Similar inequalities were developed in [ 38 , 20 , 54 ] in different settings . See [ 40 , Chapter 20 ] for an overview of the method of mixtures . We also propose KLUinf - and KL L inf -based tight anytime-valid confidence intervals for CVaR for heavy-tailed distributions , and show that classical confidence intervals derived using popular truncation-based estimators can be recovered using our method , with only a minor overhead ( see Section 4.3 ) . Since distributions in L are not characterized by parameters , we work in the space of probability measures instead of in the Euclidean space . A key and non-trivial requirement for the proof of asymptotic optimality of the algorithm is the joint continuity of KLLinf and KL U inf in a well-chosen metric , which should generate a topology that is sufficiently fine to ensure this continuity , but coarse to ensure fast convergence of the empirical distributions to the true-arm distributions . We endow P ( < ) with the topology of weak convergence , or equivalently , with the Lévy metric ( see Section 2 for definitions ) . Another nuance in our analysis is that the empirical distributions may not lie in L. This is handled by projecting these on to L under a suitable metric . Our proposed algorithm is a plugin strategy that involves solving the lower bound problem using the empirical distributions as a proxy for the actual arm distributions . This can be computationally demanding especially as the underlying samples in the empirical distribution become large . To ease the numerical burden we propose modifications that require solving the lower bound only order log ( n ) many times till stage n of the algorithm ( where n samples are generated ) . This modification substantially reduces the computation burden . We show that it is optimal up to a constant ( Appendix K ) . VaR problem : Our algorithm for CVaR , with KLLinf and KLUinf replaced by the corresponding functionals with the VaR constraints instead , is asymptotically optimal for this problem in complete generality ( Section 4.2 ) . Here , KLUinf and KL L inf have closed form representations . However , they are no longer jointly-continuous in the Lévy metric , which introduces new technical challenges in the analysis of the algorithm .

Summary: This paper addresses the multi-armed bandit best arm identification problem which aims to identify the arm with the minimum tail risk measured by VaR, CVaR, or a weighted sum of CVaR and the mean. It proposes an optimal $\delta$-correct algorithm that works with a mild restriction on the arm distributions for CVaR and works with any arm distributions for VaR. In particular, the paper develops a lower bound for the problem and shows that the proposed optimal $\delta$-correct algorithm has the sample complexity that matches the lower bound asymptotically (as $\delta \rightarrow 0$). Additionally, the paper also suggests a solution to control the trade-off between the time computational complexity and the sample complexity, which is useful in practical applications.