Off-Policy Risk Assessment in Contextual Bandits

√ n ) . 1 Introduction . Many practical tasks , including medical treatment [ 50 ] and content recommendation [ 31 ] are commonly modeled within the contextual bandits framework . In the online setting , an agent observes a context at each step and chooses among the available actions . The agent then receives a contextdependent reward corresponding to the action taken , but can not observe the rewards corresponding to alternative actions . In a healthcare setting , the observed context might be a vector capturing vital signs , lab tests , and other available data , while the action space might consist of the available treatments . The reward to optimize could be a measure of patient health or treatment response . While contextual bandits research has traditionally focused on the expected reward , stakeholders often care about other risk functionals ( parameters of the reward distribution ) that express real-world desiderata or have desirable statistical properties . For example , investors assess mutual funds via the Sharpe ratio , which normalizes returns by their variance [ 44 ] . Related works in reinforcement learning ( RL ) have sought to estimate the variance of returns [ 41 , 48 ] and to optimize the mean return under variance constraints [ 33 ] . In safety-critical and financial applications , researchers often measure the conditional value-at-risk ( CVaR ) , which captures the expected return among the lower α quantile of outcomes [ 40 , 28 ] . In an emerging line of RL works , researchers have explored other risk functionals , including cumulative prospect weighting [ 22 ] , distortion risk measures [ 13 ] , and exponential utility functions [ 16 ] . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In many real-world problems otherwise suited to the contextual bandits framework , experimentation turns out to be prohibitively expensive or unethical . In such settings , we might hope to evaluate prospective policies using the data collected under a previous policy . Formally , this problem is called off-policy evaluation , and our goal is to evaluate the performance of a target policy π using data collected under a behavior policy β . While most existing research focuses on estimating the expected value of the returns [ 20 , 19 ] , one recent paper evaluates the variance of returns [ 10 ] . In this paper , we propose practical methods and the first sample complexity guarantees for off-policy risk evaluation , addressing a diverse set of objectives of interest to researchers and practitioners . Towards this end , we introduce Lipschitz risk functionals which encompass all objectives for which the risk ( i ) depends only on the CDF of rewards ; and ( ii ) is Lipschitz with respect to changes in the CDF ( as assessed via the sup norm ) . We prove that for bounded rewards , this class subsumes many risk functionals of practical interest , including variance , mean-variance , conditional value-at-risk , and cumulative prospect weighting , among others . Thus , given accurate estimates of the CDF of rewards under π , we can accurately estimate Lipschitz risks . Moreover , ( sup norm ) bounds on our CDF estimates imply bounds on the corresponding plugin estimates for any Lipschitz risks . The key remaining step is to establish finite sample guarantees on the error in estimating the target policy ’ s CDF of rewards . Our analysis centers on an importance sampling estimator ( Section 5.1 ) , and a variance-reduced doubly robust estimator ( Section 5.3 ) . We derive finite sample concentrations for both CDF estimators , showing that they achieve the desired O ( 1/ √ n ) rates , where n is the sample size . Moreover , the estimation error for any Lipschitz risk is scales with its Lipschitz constant , and similarly converges as O ( 1/ √ n ) . We assemble these results into an algorithm called OPRA ( Algorithm 1 ) that outputs a comprehensive risk assessment for a target policy π , using any set of Lipschitz risk functionals . Notably , because all risk estimates share the same underlying CDF estimate , our error guarantees hold simultaneously for all estimated risk functionals in the set , regardless of the cardinality ( Section 6 ) . Finally , we present experiments that demonstrate the practical applicability our estimators . 2 Related Work . The study of risk functionals and risk-aware algorithms is core to the decision making literature [ 3 , 40 , 29 , 43 , 1 , 37 , 25 ] . In the bandit literature , many works address regret minimization problems using risk functionals ; popular examples include the CVaR , value-at-risk , and mean-variance [ 8 , 41 , 54 , 60 ] . [ 49 ] studies optimistic UCB exploration for optimizing CVaR while [ 11 , 6 ] study Thompson sampling , and [ 27 , 7 ] study regret minimization for linear combinations of the mean and CVaR . Using the CPT risk functional , [ 22 ] considers regret minimization in both K-armed bandits and linear contextual bandits . [ 53 , 35 ] tackle the problem of black-box function optimization under different risk functionals . In off-policy evaluation , we face an additional challenge due to the discrepancy between the data distribution and that induced by the target policy . Importance sampling ( IS ) estimators are among the most prominent methods for dealing with distribution shift [ 2 , 23 , 45 ] . Doubly robust ( DR ) estimators [ 39 , 5 ] leverage ( possibly misspecified ) models to achieve lower variance without sacrificing consistency . These estimators have been adapted for off-policy evaluation in multi-armed bandits [ 32 , 52 , 10 ] , contextual bandits [ 20 , 19 , 57 ] , and Markov decision processes [ 24 , 51 ] After deriving our key results , we learned of an independent ( also unpublished ) work [ 9 ] that also employs importance sampling to estimate CDFs for the purpose of providing off-policy estimates for parameters of the reward distribution . However , they do not establish uniform concentration of their CDF estimates or formally relate the errors in CDF and ( downstream ) parameter estimation leaving open questions concerning the convergence ( both in finite samples and asymptotically ) of the parameter estimates . Our work formulates both importance sampling and variance-reduced doubly robust estimators and provides the first uniform finite sample concentration bounds for both types of CDF and risk estimates . For empirical CDF estimation , we build on seminal work by [ 21 ] , which provided an approximation-theoretic concentration bound that was later tightened by [ 34 ] . 3 Problem Setting . We denote contexts by X and the corresponding context space by X . Similarly , we denote actions by A and the corresponding action space byA . We study the contextual bandit problem characterized by a fixed probability measure over context space X , and a conditional reward distributionR ( ·|X , A ) . In the off-policy setting , we have access to a dataset D generated using a behavior policy β that interacts with the environment for n rounds as follows : at each round , a new context X is drawn and then the policy β chooses an action A ∼ β ( ·|X ) . The environment then reveals the reward R ∼ R ( ·|X , A ) for only the chosen action A . Running this process for n stepsgenerates a dataset D : = { xi , ai , ri } ni=1 . In the off-policy evaluation setting , our goal is to evaluate the performance of a target policy π , using only a dataset D. Next , we can express our sample space in terms of the contexts , actions , and rewards : Ω = ( X ×A× R ) . Let ( Ω , F , Pβ ) be the probability space induced by the behavior policy β , and ( Ω , F , P ) the probability space induced by the target policy π . We assume that P is absolutely continuous with respect to Pβ . For any context x and action a , the importance weight expresses the ratio between the two densities w ( ω ) = w ( a , x ) = β ( a|x ) π ( a|x ) , and the maximum weight wmax = supa , x w ( a , x ) is simply the supremum taken over all contexts and actions . Further , let w2 = EPβ [ w ( A , X ) 2 ] denote the exponential of the second order Rényi divergence . Note that by definition , w2 ≤ wmax , and in practice , we often have w2 wmax . Finally , we introduce some notation for describing CDFs : For any t ∈ R , let F ( t ) = EP [ 1 { R≤t } ] denote the CDF under the target policy ; further , let G ( t ; X , A ) = EP [ 1 { R≤t } |X , A ] = EPβ [ 1 { R≤t } |X , A ] denote the CDF of rewards conditioned on a context X and action A , which is independent of the policy . Lastly , for any t ∈ R , we denote the variance by σ2 ( t ; X , A ) = VP [ 1 { R≤t } |X , A ] = VPβ [ 1 { R≤t } |X , A ] . 4 Lipschitz Risk Functionals . We now introduce Lipschitz risk functionals , a novel class of objectives for which absolute differences in the risk are bounded by sup norm differences in the CDF of rewards . After formally defining the class , we provide an in-depth review of common risk functionals and their relationship to the CDF of rewards . When possible , we derive the associated Lipschitz constants , when rewards are bounded on support [ 0 , D ] , relegating all proofs to Appendix A.2 . 4.1 Defining the Lipschitz Risk Functionals . The Lipschitz risk functionals are a subset of the broader family of law-invariant risk functionals . Formally , let Z ∈ L∞ ( Ω , FZ , PZ ) denote a real-valued random variable that admits a CDF FZ ∈ L∞ ( R , B ( R ) ) . A risk functional ρ is a mapping from a space of random variables to the space of real numbers ρ : L∞ ( Ω , FZ , PZ ) → R. Any risk functional ρ is said to be law-invariant if ρ ( Z ) depends only on the distribution of Z [ 30 ] . Definition 4.1 ( Law-Invariant Risk Functional ) . A risk functional ρ : L∞ ( Ω , F , P ) → R , is lawinvariant if for any pair of random variables Z and Z ′ , FZ = FZ′ =⇒ ρ ( Z ) = ρ ( Z ′ ) . When clear from the context , we sometimes abuse notation by writing ρ ( FZ ) in place of ρ ( Z ) . In general , it may not be practical to estimate risk functionals that are not law invariant from data [ 4 ] . Thus focusing on law-invariant risks is only mildly restrictive . We can now formally define the Lipschitz risk functionals : Definition 4.2 ( Lipschitz Risk Functional ) . A law invariant risk functional ρ is L-Lipschitz if for any pair of CDFs FZ and FZ′ and some L ∈ ( 0 , ∞ ) , it satisfies |ρ ( FZ ) − ρ ( FZ′ ) | ≤ L‖FZ − FZ′‖∞ . A risk functional is L-Lipschitz if , for any two random variables Z , Z ′ , its value is upper bounded by the sup-norm of the difference between their corresponding CDFs . The significance of this Lipschitzness property in the contextual bandit setting is that , given a high confidence bound on the error of the estimated CDF of rewards for a policy π , we can obtain a high confidence bound on its evaluation under any L-Lipschitz law-invariant risk functional on the distribution of rewards .

Summary: The paper proposes off-policy risk assessment(OPRA), a framework that estimate a risk functional through first estimate CDF and second plugin estimates for any risks functional that satisfies Lipschitz properties. The new proposed framework is elegant and can simultaneously estimate a broad class  of objectives including CVar, variance, mean-variance and many others. The paper also provides theoretical convergence analysis for the estimator.