Off-Policy Risk Assessment in Contextual Bandits

√ n ) . 1 Introduction . Many practical tasks , including medical treatment [ 50 ] and content recommendation [ 31 ] are commonly modeled within the contextual bandits framework . In the online setting , an agent observes a context at each step and chooses among the available actions . The agent then receives a contextdependent reward corresponding to the action taken , but can not observe the rewards corresponding to alternative actions . In a healthcare setting , the observed context might be a vector capturing vital signs , lab tests , and other available data , while the action space might consist of the available treatments . The reward to optimize could be a measure of patient health or treatment response . While contextual bandits research has traditionally focused on the expected reward , stakeholders often care about other risk functionals ( parameters of the reward distribution ) that express real-world desiderata or have desirable statistical properties . For example , investors assess mutual funds via the Sharpe ratio , which normalizes returns by their variance [ 44 ] . Related works in reinforcement learning ( RL ) have sought to estimate the variance of returns [ 41 , 48 ] and to optimize the mean return under variance constraints [ 33 ] . In safety-critical and financial applications , researchers often measure the conditional value-at-risk ( CVaR ) , which captures the expected return among the lower α quantile of outcomes [ 40 , 28 ] . In an emerging line of RL works , researchers have explored other risk functionals , including cumulative prospect weighting [ 22 ] , distortion risk measures [ 13 ] , and exponential utility functions [ 16 ] . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In many real-world problems otherwise suited to the contextual bandits framework , experimentation turns out to be prohibitively expensive or unethical . In such settings , we might hope to evaluate prospective policies using the data collected under a previous policy . Formally , this problem is called off-policy evaluation , and our goal is to evaluate the performance of a target policy π using data collected under a behavior policy β . While most existing research focuses on estimating the expected value of the returns [ 20 , 19 ] , one recent paper evaluates the variance of returns [ 10 ] . In this paper , we propose practical methods and the first sample complexity guarantees for off-policy risk evaluation , addressing a diverse set of objectives of interest to researchers and practitioners . Towards this end , we introduce Lipschitz risk functionals which encompass all objectives for which the risk ( i ) depends only on the CDF of rewards ; and ( ii ) is Lipschitz with respect to changes in the CDF ( as assessed via the sup norm ) . We prove that for bounded rewards , this class subsumes many risk functionals of practical interest , including variance , mean-variance , conditional value-at-risk , and cumulative prospect weighting , among others . Thus , given accurate estimates of the CDF of rewards under π , we can accurately estimate Lipschitz risks . Moreover , ( sup norm ) bounds on our CDF estimates imply bounds on the corresponding plugin estimates for any Lipschitz risks . The key remaining step is to establish finite sample guarantees on the error in estimating the target policy ’ s CDF of rewards . Our analysis centers on an importance sampling estimator ( Section 5.1 ) , and a variance-reduced doubly robust estimator ( Section 5.3 ) . We derive finite sample concentrations for both CDF estimators , showing that they achieve the desired O ( 1/ √ n ) rates , where n is the sample size . Moreover , the estimation error for any Lipschitz risk is scales with its Lipschitz constant , and similarly converges as O ( 1/ √ n ) . We assemble these results into an algorithm called OPRA ( Algorithm 1 ) that outputs a comprehensive risk assessment for a target policy π , using any set of Lipschitz risk functionals . Notably , because all risk estimates share the same underlying CDF estimate , our error guarantees hold simultaneously for all estimated risk functionals in the set , regardless of the cardinality ( Section 6 ) . Finally , we present experiments that demonstrate the practical applicability our estimators . 2 Related Work . The study of risk functionals and risk-aware algorithms is core to the decision making literature [ 3 , 40 , 29 , 43 , 1 , 37 , 25 ] . In the bandit literature , many works address regret minimization problems using risk functionals ; popular examples include the CVaR , value-at-risk , and mean-variance [ 8 , 41 , 54 , 60 ] . [ 49 ] studies optimistic UCB exploration for optimizing CVaR while [ 11 , 6 ] study Thompson sampling , and [ 27 , 7 ] study regret minimization for linear combinations of the mean and CVaR . Using the CPT risk functional , [ 22 ] considers regret minimization in both K-armed bandits and linear contextual bandits . [ 53 , 35 ] tackle the problem of black-box function optimization under different risk functionals . In off-policy evaluation , we face an additional challenge due to the discrepancy between the data distribution and that induced by the target policy . Importance sampling ( IS ) estimators are among the most prominent methods for dealing with distribution shift [ 2 , 23 , 45 ] . Doubly robust ( DR ) estimators [ 39 , 5 ] leverage ( possibly misspecified ) models to achieve lower variance without sacrificing consistency . These estimators have been adapted for off-policy evaluation in multi-armed bandits [ 32 , 52 , 10 ] , contextual bandits [ 20 , 19 , 57 ] , and Markov decision processes [ 24 , 51 ] After deriving our key results , we learned of an independent ( also unpublished ) work [ 9 ] that also employs importance sampling to estimate CDFs for the purpose of providing off-policy estimates for parameters of the reward distribution . However , they do not establish uniform concentration of their CDF estimates or formally relate the errors in CDF and ( downstream ) parameter estimation leaving open questions concerning the convergence ( both in finite samples and asymptotically ) of the parameter estimates . Our work formulates both importance sampling and variance-reduced doubly robust estimators and provides the first uniform finite sample concentration bounds for both types of CDF and risk estimates . For empirical CDF estimation , we build on seminal work by [ 21 ] , which provided an approximation-theoretic concentration bound that was later tightened by [ 34 ] . 3 Problem Setting . We denote contexts by X and the corresponding context space by X . Similarly , we denote actions by A and the corresponding action space byA . We study the contextual bandit problem characterized by a fixed probability measure over context space X , and a conditional reward distributionR ( ·|X , A ) . In the off-policy setting , we have access to a dataset D generated using a behavior policy β that interacts with the environment for n rounds as follows : at each round , a new context X is drawn and then the policy β chooses an action A ∼ β ( ·|X ) . The environment then reveals the reward R ∼ R ( ·|X , A ) for only the chosen action A . Running this process for n stepsgenerates a dataset D : = { xi , ai , ri } ni=1 . In the off-policy evaluation setting , our goal is to evaluate the performance of a target policy π , using only a dataset D. Next , we can express our sample space in terms of the contexts , actions , and rewards : Ω = ( X ×A× R ) . Let ( Ω , F , Pβ ) be the probability space induced by the behavior policy β , and ( Ω , F , P ) the probability space induced by the target policy π . We assume that P is absolutely continuous with respect to Pβ . For any context x and action a , the importance weight expresses the ratio between the two densities w ( ω ) = w ( a , x ) = β ( a|x ) π ( a|x ) , and the maximum weight wmax = supa , x w ( a , x ) is simply the supremum taken over all contexts and actions . Further , let w2 = EPβ [ w ( A , X ) 2 ] denote the exponential of the second order Rényi divergence . Note that by definition , w2 ≤ wmax , and in practice , we often have w2 wmax . Finally , we introduce some notation for describing CDFs : For any t ∈ R , let F ( t ) = EP [ 1 { R≤t } ] denote the CDF under the target policy ; further , let G ( t ; X , A ) = EP [ 1 { R≤t } |X , A ] = EPβ [ 1 { R≤t } |X , A ] denote the CDF of rewards conditioned on a context X and action A , which is independent of the policy . Lastly , for any t ∈ R , we denote the variance by σ2 ( t ; X , A ) = VP [ 1 { R≤t } |X , A ] = VPβ [ 1 { R≤t } |X , A ] . 4 Lipschitz Risk Functionals . We now introduce Lipschitz risk functionals , a novel class of objectives for which absolute differences in the risk are bounded by sup norm differences in the CDF of rewards . After formally defining the class , we provide an in-depth review of common risk functionals and their relationship to the CDF of rewards . When possible , we derive the associated Lipschitz constants , when rewards are bounded on support [ 0 , D ] , relegating all proofs to Appendix A.2 . 4.1 Defining the Lipschitz Risk Functionals . The Lipschitz risk functionals are a subset of the broader family of law-invariant risk functionals . Formally , let Z ∈ L∞ ( Ω , FZ , PZ ) denote a real-valued random variable that admits a CDF FZ ∈ L∞ ( R , B ( R ) ) . A risk functional ρ is a mapping from a space of random variables to the space of real numbers ρ : L∞ ( Ω , FZ , PZ ) → R. Any risk functional ρ is said to be law-invariant if ρ ( Z ) depends only on the distribution of Z [ 30 ] . Definition 4.1 ( Law-Invariant Risk Functional ) . A risk functional ρ : L∞ ( Ω , F , P ) → R , is lawinvariant if for any pair of random variables Z and Z ′ , FZ = FZ′ =⇒ ρ ( Z ) = ρ ( Z ′ ) . When clear from the context , we sometimes abuse notation by writing ρ ( FZ ) in place of ρ ( Z ) . In general , it may not be practical to estimate risk functionals that are not law invariant from data [ 4 ] . Thus focusing on law-invariant risks is only mildly restrictive . We can now formally define the Lipschitz risk functionals : Definition 4.2 ( Lipschitz Risk Functional ) . A law invariant risk functional ρ is L-Lipschitz if for any pair of CDFs FZ and FZ′ and some L ∈ ( 0 , ∞ ) , it satisfies |ρ ( FZ ) − ρ ( FZ′ ) | ≤ L‖FZ − FZ′‖∞ . A risk functional is L-Lipschitz if , for any two random variables Z , Z ′ , its value is upper bounded by the sup-norm of the difference between their corresponding CDFs . The significance of this Lipschitzness property in the contextual bandit setting is that , given a high confidence bound on the error of the estimated CDF of rewards for a policy π , we can obtain a high confidence bound on its evaluation under any L-Lipschitz law-invariant risk functional on the distribution of rewards .

Summary: Previous OPE studies mostly are focused on the overall expected return. This paper studies off-policy evaluation for a more broad class of objectives, i.e., the Lipschitz risk functionals, including CVaR, variance, mean-variance, etc. The main idea is to first use an IPS/DR estimate to estimate the target policy's CDF, then use it as a plug-in estimator to any functionals that satisfy the Lipschitz property. Theoretically, an $O(1/\sqrt{n})$ error bound is derived for the Lipschitz estimate. Empirically, the performance of the plug-in estimator (based on different CDF estimates) is examined.  OPE is crucial to estimate the performance of any new policy while online A/B testing or experimentation might be infeasible. Most prior works on OPE focus on efficient estimation of the expected reward of the policy, while this paper studies the estimation problem under more general metrics, which may be of high importance in real-world applications when people care about different functionals, such as CVaR in finance.