Post-Contextual-Bandit Inference

1 Introduction . Contextual bandits , where personalized decisions are made sequentially and simultaneously with data collection , are increasingly used to address important decision-making problems where data is limited and/or expensive to collect , with applications in product recommendation [ Li et al. , 2010 ] , revenue management [ Kallus and Udell , 2020 , Qiang and Bayati , 2016 ] , and personalized medicine [ Tewari and Murphy , 2017 ] . Adaptive experiments , whether based on bandit algorithms or Bayesian optimization , are increasingly being considered in place of classic randomized trials in order to improve both the outcomes for study participants and the chance of identifying the best treatment allocations [ Athey et al. , 2018 , Quinn et al. , 2019 , Kasy and Sautmann , 2021 , Bakshy et al. , 2018 ] . But , at the end of the study , we still want to construct valid confidence intervals on average treatment effects , subgroup effects , or the value of new personalized interventions . Such confidence intervals are , for example , crucial for enabling credible inference on the presence or absence of improvement of novel policies . However , due to the adaptive nature of the data collection , unlike classic randomized trials , standard estimates and their confidence intervals actually fail to provide correct coverage , that is , contain the true parameter with the desired confidence probability ( e.g. , 95 % ) . A variety of recent work has recognized this and offered remedies [ Hadad et al. , 2019 , Luedtke and van der Laan , 2016 ] , but only for the case of non-contextual adaptive data collection . Like classic 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . confidence intervals , when data comes from a contextual bandit – or any other context-dependent adaptive data collection – these intervals also fail to provide correct coverage . In this paper , we propose a novel asymptotically normal estimator for the value of a ( possibly contextual ) policy from context-dependent adaptively collected data . This asymptotic normality leads directly to the construction of valid confidence intervals . Our estimator takes the form of a stabilized doubly robust estimator , that is , a weighted time average of an estimate of the so-called canonical gradient using plug in estimators for the outcome model , where each time point is inversely weighted by its estimated conditional standard deviation given the past . We term this the Contextual Adaptive Doubly Robust ( CADR ) estimator . We show that , given consistent conditional variance estimates which at each time point only depend on previous data , the CADR estimator is asymptotically normal , and as a result we can easily construct asymptotically valid confidence intervals . This normality is in fact robust to misspecifying the outcome model . A significant technical challenge is actually constructing such variance estimators . We resolve this using an adaptive variance estimator based on the importance-sampling ratio of current to past ( adaptive ) policies at each time point . We also show that we can reliably estimate outcome models from the adaptively-collected data so that we can plug them in . Extensive experiments using 57 OpenML datasets demonstrate the failure of previous approaches and the success of ours at constructing confidence intervals with correct coverage . 1.1 Problem Statement and Notation . The data . Our data consists of a sequence of observations indexed t = 1 , . . . , T comprising of context X ( t ) ∈ X , action A ( t ) ∈ A , and outcome Y ( t ) ∈ Y ⊂ R generated by an adaptive experiment , such as a contextual bandit algorithm . Roughly , at each round t = 1 , 2 , . . . , T , an agent formed a contextual policy gt ( a | x ) based on all past observations , then observed an independently drawn context vector X ( t ) ∼ Q0 , X , carried out an action A ( t ) drawn from its current policy gt ( · | X ( t ) ) , and observed an outcome Y ( t ) ∼ Q0 , Y ( · | A ( t ) , X ( t ) ) depending only on the present context and action . The action and context measurable spaces X , A are arbitrary , e.g. , finite or continuous . We assume outcomes are bounded and , without loss of generality , take Y = [ −1 , 1 ] . More formally , we let O ( t ) : = ( X ( t ) , A ( t ) , Y ( t ) ) and make the following assumptions about the sequence O ( 1 ) , . . . , O ( T ) comprising our dataset . First , we assume X ( t ) is independent of all else given A ( t ) and has a time-independent marginal distribution that we denote by Q0 , X . Second , we assume A ( t ) is independent of all else given O ( 1 ) , . . . , O ( t − 1 ) , X ( t ) and we set gt ( · | X ( t ) ) to its ( random ) conditional distribution given O ( 1 ) , . . . , O ( t − 1 ) , X ( t ) . Third , we assume Y ( t ) is independent of all else given X ( t ) , A ( t ) and has a time-independent conditional distribution given X ( t ) = x , A ( t ) = a that is denoted by Q0 , Y ( · | A , X ) . The distributions Q0 , X and Q0 , Y are unknown , while the policies gt ( a | x ) are known , as would be the case when running an adaptive experiment . To simplify presentation we endow A with a base measure µA ( e.g. , counting for finite actions or Lebesgue for continuous actions ) and identify policies gt with conditional densities with respect to ( w.r.t . ) µA . In the case of K < ∞ actions , policies are maps from X to the K-simplex . Note that , as the agent updates its policy based on already collected observations , gt is a random O ( 1 ) , . . . , O ( t − 1 ) -measurable object . This is the major departure from the setting considered in other literature on off-policy evaluation , which only consider a fixed logging policy , gt = g , that is independent of the data . See Section 1.2 . The target parameter . We are interested in inference on a generalized average causal effect expressed as a functional of the unknown distributions above , Ψ0 = Ψ ( Q0 , X , Q0 , Y ) , where for any distributions QX , QY , we define Ψ ( QX , QY ) : = ∫ yQX ( dx ) g ∗ ( a | x ) dµA ( a ) QY ( dy | a , x ) , where g∗ ( a | x ) : A × X → [ −G , G ] is a given fixed , bounded function . Two examples are : ( a ) when g∗ is a policy ( conditional density ) , then Ψ0 is its value ; ( b ) when g∗ is the difference between two policies then Ψ0 is the difference between their values . A prominent example of the latter is when A = { +1 , −1 } and g∗ ( a | x ) = a , which is known as the average treatment effect . If we include an indicator for x being in some set , then we get the subgroup effect . Defining the conditional mean outcome , Q̄0 ( a , x ) : = EQ0 , Y ( ·|x , a ) [ Y ] = ∫ yQ0 , Y ( dy | a , x ) , we note that the target parameter only depends on Q0 , Y via Q̄0 , so we also overload notation and write Ψ ( QX , Q̄ ) = ∫ Q̄ ( a , x ) QX ( dx ) g ( a | x ) dµA ( a ) for any function Q̄ : A×X → Y . Note that when |A| < ∞ and µA is the counting measure , the integral over a is a simple sum . Canonical gradient . We will make repeated use of the following function : for any conditional density ( a , x ) 7→ g ( a | x ) , any probability distribution QX over the context space X , and any function Q̄ : A×X , we define the function D′ ( g , Q̄ ) : O → R by D′ ( g , Q̄ ) ( x , a , y ) : = g∗ ( a | x ) g ( a | x ) ( y − Q̄ ( a , x ) ) + ∫ Q̄ ( a′ , x ) g∗ ( a′ | x ) dµA ( a′ ) . Further , define D ( g , QX , Q̄ ) = D′ ( g , QX , Q̄ ) − Ψ ( QX , Q̄ ) , which coincides with the so-called canonical gradient of the target parameter Ψ w.r.t . the usual nonparametric statistical model comprising all joint distributions over O [ van der Vaart , 2000 , van der Laan and Robins , 2003 ] . Integration operator notation . For any policy g and distributions QX , QY , denote by PQ , g the induced distribution on O : = X × A × Y . For any function f : O → R , we use the integration operator notation PQ , gf = ∫ f ( x , a , y ) QX ( dx ) g ( a | x ) dµA ( a ) QY ( dy | a , x ) , that is , the expectation w.r.t . PQ , g alone . Then , for example , for anyO ( 1 ) , . . . , O ( s−1 ) -measurable random function f : O → R , we have that PQ0 , gsf = EQ0 , gs [ f ( O ( s ) ) | O ( 1 ) , . . . , O ( s− 1 ) ] . 1.2 Related Literature and Challenges for Post-Contextual-Bandit Inference . Off-policy evaluation . In non-adaptive settings , where gt = g is fixed and does not depend on previous observations , common off-the shelf estimators for the mean outcome under g∗ include the Inverse Propensity Scoring ( IPS ) estimator [ Beygelzimer and Langford , 2009 , Li et al. , 2011 ] and and the Doubly Robust ( DR ) estimator [ Dudı́k et al. , 2011 , Robins et al. , 1994 ] : Ψ̂IPS : = 1 T T∑ t=1 D′ ( g , 0 ) , Ψ̂DR : = 1 T T∑ t=1 D′ ( g , ̂̄Q ) where ̂̄Q is an estimator of the outcome model Q̄0 ( a , x ) . If we use cross-fitting to estimate ̂̄Q [ Chernozhukov et al. , 2018 ] , then both the IPS and DR estimators are unbiased and asymptotically normal , permitting straightforward inference using Wald confidence intervals ( i.e. , ±1.96 of the estimated standard error ) . There also exist many variants of the IPS and DR estimators that , rather than plugging in the importance sampling ( IS ) ratios ( g∗/gt ) ( A ( t ) | X ( t ) ) and/or outcome-model estimators , instead choose them directly with the aim to minimize error [ e.g . Kallus , 2018 , Farajtabar et al. , 2018 , Thomas and Brunskill , 2016 , Wang et al. , 2017 , Kallus and Uehara , 2019 ] . Inference challenges in adaptive settings . In the adaptive setting , it is easy to see that , if in the tth term for DR we use an outcome model ̂̄Qt−1 fit using only the observations O ( 1 ) , . . . , O ( t− 1 ) , then both the IPS and DR estimators both remain unbiased . However , neither generally converges to a normal distribution . One key difference between the non-adaptive and adaptive settings is that the IS ratios ( g∗/gt ) ( A ( t ) | X ( t ) ) can both diverge to infinity or converge to zero . As a result of this , the above two estimators may either be dominated by their first terms or their last terms . At a more theoretical level , this violates the classical condition of martingale central limit theorems that the conditional variance of the terms given previous observations stabilizes asymptotically . Stabilized estimators . The issue for inference due to instability of the DR estimator terms was recognized by Luedtke and van der Laan [ 2016 ] in another setting . They work in the non-adaptive setting but consider the problem of inferring the maximum mean outcome over all policies when the optimal policy is non-unique . Their proposal is a so-called stabilized estimator , in which each term is inversely weighted by an estimate of its conditional standard deviation given the previous terms . This stabilization trick has been also been reused for off-policy inference from non-contextual bandit data by Hadad et al . [ 2019 ] , as the stabilized estimator remains asymptotically normal , permitting inference . In their non-contextual setting , an estimate of the conditional standard deviation of the terms can easily be obtained by the inverse square root propensities . In contrast , in our contextual setting , obtaining valid stabilization weights is more challenging and requires a construction involving adaptive training on past data . Zhang et al . [ 2021 ] use similar weights to Hadad et al . [ 2019 ] in order to stabilize an M -estimator fit to adaptively collected data for the purpose of inference on a true model for Q̄0 . While they do consider a contextual setting , their method requires correct specification , else the inferential target depends on the logging policies . We focus only on off-policy evaluation but do not require well-specified models . Concurrent work of Zhan et al . [ 2021 ] . The paper of Zhan et al . [ 2021 ] , which was published after the submission of our paper , is also motivated by establishing an unbiased and asymptotically normal estimator from data collected by a contextual bandit . Both our method and theirs rely on adaptively weighted DR scores and yield , under certain conditions , asymptotically normal estimators . However , the way they construct their adaptive weights is quite different from ours . In particular , their adaptive weights at time t ( a ) do not incorporate an outcome model and do not rely on it and ( b ) depend only on the known propensity score , which is plugged-in . As a result of these different constructions , our analysis and theirs require different sets of conditions . While we require slightly more stringent conditions on the exploration rate and the complexity of the logging policy class than they do ( our assumptions 5 and 6 ) , they require some form of asymptotic stabilization of the logging policies ( their conditions ( 8 ) and ( 12 ) ) . An instance where the logging policy does not stabilize is , for example , when various actions have the same expected reward on certain subsets of the context space . Our approach in this paper is to use the variance-stabilization weights for the contextual bandit setting , just like Hadad et al . [ 2019 ] do the in the non-contextual bandit setting and Luedtke and van der Laan [ 2016 ] do in an iid setting with multiple optimal policies . Although Zhan et al . [ 2021 ] also tackle the contextual bandit setting , they do not use the variance-stabilization weights ; correspondingly their conditions and results apply in different , complementary settings . The key challenge we tackle in applying variance-stabilization weights in the contextual bandit setting is dealing with the need to use sequentially estimated weights .

Summary: This study proposes a variance-stabilized estimator for policy evaluation from data obtained from contextual bandit algorithms, which is inspired by [Luedtke and van der Laan 2016]  and [Hadad et al. 2019].  In this study, the authors estimate the variance of the score function (canonical gradient) from samples, which standardizes the MDS for applying martingale CLT. [Luedtke, and van der Laan 2016] Statistical inference for the mean outcome under a possibly non-unique optimal treatment strategy   [Hadad et al 2019] Confidence Intervals for Policy Evaluation in Adaptive Experiments.