Post-Contextual-Bandit Inference

1 Introduction . Contextual bandits , where personalized decisions are made sequentially and simultaneously with data collection , are increasingly used to address important decision-making problems where data is limited and/or expensive to collect , with applications in product recommendation [ Li et al. , 2010 ] , revenue management [ Kallus and Udell , 2020 , Qiang and Bayati , 2016 ] , and personalized medicine [ Tewari and Murphy , 2017 ] . Adaptive experiments , whether based on bandit algorithms or Bayesian optimization , are increasingly being considered in place of classic randomized trials in order to improve both the outcomes for study participants and the chance of identifying the best treatment allocations [ Athey et al. , 2018 , Quinn et al. , 2019 , Kasy and Sautmann , 2021 , Bakshy et al. , 2018 ] . But , at the end of the study , we still want to construct valid confidence intervals on average treatment effects , subgroup effects , or the value of new personalized interventions . Such confidence intervals are , for example , crucial for enabling credible inference on the presence or absence of improvement of novel policies . However , due to the adaptive nature of the data collection , unlike classic randomized trials , standard estimates and their confidence intervals actually fail to provide correct coverage , that is , contain the true parameter with the desired confidence probability ( e.g. , 95 % ) . A variety of recent work has recognized this and offered remedies [ Hadad et al. , 2019 , Luedtke and van der Laan , 2016 ] , but only for the case of non-contextual adaptive data collection . Like classic 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . confidence intervals , when data comes from a contextual bandit – or any other context-dependent adaptive data collection – these intervals also fail to provide correct coverage . In this paper , we propose a novel asymptotically normal estimator for the value of a ( possibly contextual ) policy from context-dependent adaptively collected data . This asymptotic normality leads directly to the construction of valid confidence intervals . Our estimator takes the form of a stabilized doubly robust estimator , that is , a weighted time average of an estimate of the so-called canonical gradient using plug in estimators for the outcome model , where each time point is inversely weighted by its estimated conditional standard deviation given the past . We term this the Contextual Adaptive Doubly Robust ( CADR ) estimator . We show that , given consistent conditional variance estimates which at each time point only depend on previous data , the CADR estimator is asymptotically normal , and as a result we can easily construct asymptotically valid confidence intervals . This normality is in fact robust to misspecifying the outcome model . A significant technical challenge is actually constructing such variance estimators . We resolve this using an adaptive variance estimator based on the importance-sampling ratio of current to past ( adaptive ) policies at each time point . We also show that we can reliably estimate outcome models from the adaptively-collected data so that we can plug them in . Extensive experiments using 57 OpenML datasets demonstrate the failure of previous approaches and the success of ours at constructing confidence intervals with correct coverage . 1.1 Problem Statement and Notation . The data . Our data consists of a sequence of observations indexed t = 1 , . . . , T comprising of context X ( t ) ∈ X , action A ( t ) ∈ A , and outcome Y ( t ) ∈ Y ⊂ R generated by an adaptive experiment , such as a contextual bandit algorithm . Roughly , at each round t = 1 , 2 , . . . , T , an agent formed a contextual policy gt ( a | x ) based on all past observations , then observed an independently drawn context vector X ( t ) ∼ Q0 , X , carried out an action A ( t ) drawn from its current policy gt ( · | X ( t ) ) , and observed an outcome Y ( t ) ∼ Q0 , Y ( · | A ( t ) , X ( t ) ) depending only on the present context and action . The action and context measurable spaces X , A are arbitrary , e.g. , finite or continuous . We assume outcomes are bounded and , without loss of generality , take Y = [ −1 , 1 ] . More formally , we let O ( t ) : = ( X ( t ) , A ( t ) , Y ( t ) ) and make the following assumptions about the sequence O ( 1 ) , . . . , O ( T ) comprising our dataset . First , we assume X ( t ) is independent of all else given A ( t ) and has a time-independent marginal distribution that we denote by Q0 , X . Second , we assume A ( t ) is independent of all else given O ( 1 ) , . . . , O ( t − 1 ) , X ( t ) and we set gt ( · | X ( t ) ) to its ( random ) conditional distribution given O ( 1 ) , . . . , O ( t − 1 ) , X ( t ) . Third , we assume Y ( t ) is independent of all else given X ( t ) , A ( t ) and has a time-independent conditional distribution given X ( t ) = x , A ( t ) = a that is denoted by Q0 , Y ( · | A , X ) . The distributions Q0 , X and Q0 , Y are unknown , while the policies gt ( a | x ) are known , as would be the case when running an adaptive experiment . To simplify presentation we endow A with a base measure µA ( e.g. , counting for finite actions or Lebesgue for continuous actions ) and identify policies gt with conditional densities with respect to ( w.r.t . ) µA . In the case of K < ∞ actions , policies are maps from X to the K-simplex . Note that , as the agent updates its policy based on already collected observations , gt is a random O ( 1 ) , . . . , O ( t − 1 ) -measurable object . This is the major departure from the setting considered in other literature on off-policy evaluation , which only consider a fixed logging policy , gt = g , that is independent of the data . See Section 1.2 . The target parameter . We are interested in inference on a generalized average causal effect expressed as a functional of the unknown distributions above , Ψ0 = Ψ ( Q0 , X , Q0 , Y ) , where for any distributions QX , QY , we define Ψ ( QX , QY ) : = ∫ yQX ( dx ) g ∗ ( a | x ) dµA ( a ) QY ( dy | a , x ) , where g∗ ( a | x ) : A × X → [ −G , G ] is a given fixed , bounded function . Two examples are : ( a ) when g∗ is a policy ( conditional density ) , then Ψ0 is its value ; ( b ) when g∗ is the difference between two policies then Ψ0 is the difference between their values . A prominent example of the latter is when A = { +1 , −1 } and g∗ ( a | x ) = a , which is known as the average treatment effect . If we include an indicator for x being in some set , then we get the subgroup effect . Defining the conditional mean outcome , Q̄0 ( a , x ) : = EQ0 , Y ( ·|x , a ) [ Y ] = ∫ yQ0 , Y ( dy | a , x ) , we note that the target parameter only depends on Q0 , Y via Q̄0 , so we also overload notation and write Ψ ( QX , Q̄ ) = ∫ Q̄ ( a , x ) QX ( dx ) g ( a | x ) dµA ( a ) for any function Q̄ : A×X → Y . Note that when |A| < ∞ and µA is the counting measure , the integral over a is a simple sum . Canonical gradient . We will make repeated use of the following function : for any conditional density ( a , x ) 7→ g ( a | x ) , any probability distribution QX over the context space X , and any function Q̄ : A×X , we define the function D′ ( g , Q̄ ) : O → R by D′ ( g , Q̄ ) ( x , a , y ) : = g∗ ( a | x ) g ( a | x ) ( y − Q̄ ( a , x ) ) + ∫ Q̄ ( a′ , x ) g∗ ( a′ | x ) dµA ( a′ ) . Further , define D ( g , QX , Q̄ ) = D′ ( g , QX , Q̄ ) − Ψ ( QX , Q̄ ) , which coincides with the so-called canonical gradient of the target parameter Ψ w.r.t . the usual nonparametric statistical model comprising all joint distributions over O [ van der Vaart , 2000 , van der Laan and Robins , 2003 ] . Integration operator notation . For any policy g and distributions QX , QY , denote by PQ , g the induced distribution on O : = X × A × Y . For any function f : O → R , we use the integration operator notation PQ , gf = ∫ f ( x , a , y ) QX ( dx ) g ( a | x ) dµA ( a ) QY ( dy | a , x ) , that is , the expectation w.r.t . PQ , g alone . Then , for example , for anyO ( 1 ) , . . . , O ( s−1 ) -measurable random function f : O → R , we have that PQ0 , gsf = EQ0 , gs [ f ( O ( s ) ) | O ( 1 ) , . . . , O ( s− 1 ) ] . 1.2 Related Literature and Challenges for Post-Contextual-Bandit Inference . Off-policy evaluation . In non-adaptive settings , where gt = g is fixed and does not depend on previous observations , common off-the shelf estimators for the mean outcome under g∗ include the Inverse Propensity Scoring ( IPS ) estimator [ Beygelzimer and Langford , 2009 , Li et al. , 2011 ] and and the Doubly Robust ( DR ) estimator [ Dudı́k et al. , 2011 , Robins et al. , 1994 ] : Ψ̂IPS : = 1 T T∑ t=1 D′ ( g , 0 ) , Ψ̂DR : = 1 T T∑ t=1 D′ ( g , ̂̄Q ) where ̂̄Q is an estimator of the outcome model Q̄0 ( a , x ) . If we use cross-fitting to estimate ̂̄Q [ Chernozhukov et al. , 2018 ] , then both the IPS and DR estimators are unbiased and asymptotically normal , permitting straightforward inference using Wald confidence intervals ( i.e. , ±1.96 of the estimated standard error ) . There also exist many variants of the IPS and DR estimators that , rather than plugging in the importance sampling ( IS ) ratios ( g∗/gt ) ( A ( t ) | X ( t ) ) and/or outcome-model estimators , instead choose them directly with the aim to minimize error [ e.g . Kallus , 2018 , Farajtabar et al. , 2018 , Thomas and Brunskill , 2016 , Wang et al. , 2017 , Kallus and Uehara , 2019 ] . Inference challenges in adaptive settings . In the adaptive setting , it is easy to see that , if in the tth term for DR we use an outcome model ̂̄Qt−1 fit using only the observations O ( 1 ) , . . . , O ( t− 1 ) , then both the IPS and DR estimators both remain unbiased . However , neither generally converges to a normal distribution . One key difference between the non-adaptive and adaptive settings is that the IS ratios ( g∗/gt ) ( A ( t ) | X ( t ) ) can both diverge to infinity or converge to zero . As a result of this , the above two estimators may either be dominated by their first terms or their last terms . At a more theoretical level , this violates the classical condition of martingale central limit theorems that the conditional variance of the terms given previous observations stabilizes asymptotically . Stabilized estimators . The issue for inference due to instability of the DR estimator terms was recognized by Luedtke and van der Laan [ 2016 ] in another setting . They work in the non-adaptive setting but consider the problem of inferring the maximum mean outcome over all policies when the optimal policy is non-unique . Their proposal is a so-called stabilized estimator , in which each term is inversely weighted by an estimate of its conditional standard deviation given the previous terms . This stabilization trick has been also been reused for off-policy inference from non-contextual bandit data by Hadad et al . [ 2019 ] , as the stabilized estimator remains asymptotically normal , permitting inference . In their non-contextual setting , an estimate of the conditional standard deviation of the terms can easily be obtained by the inverse square root propensities . In contrast , in our contextual setting , obtaining valid stabilization weights is more challenging and requires a construction involving adaptive training on past data . Zhang et al . [ 2021 ] use similar weights to Hadad et al . [ 2019 ] in order to stabilize an M -estimator fit to adaptively collected data for the purpose of inference on a true model for Q̄0 . While they do consider a contextual setting , their method requires correct specification , else the inferential target depends on the logging policies . We focus only on off-policy evaluation but do not require well-specified models . Concurrent work of Zhan et al . [ 2021 ] . The paper of Zhan et al . [ 2021 ] , which was published after the submission of our paper , is also motivated by establishing an unbiased and asymptotically normal estimator from data collected by a contextual bandit . Both our method and theirs rely on adaptively weighted DR scores and yield , under certain conditions , asymptotically normal estimators . However , the way they construct their adaptive weights is quite different from ours . In particular , their adaptive weights at time t ( a ) do not incorporate an outcome model and do not rely on it and ( b ) depend only on the known propensity score , which is plugged-in . As a result of these different constructions , our analysis and theirs require different sets of conditions . While we require slightly more stringent conditions on the exploration rate and the complexity of the logging policy class than they do ( our assumptions 5 and 6 ) , they require some form of asymptotic stabilization of the logging policies ( their conditions ( 8 ) and ( 12 ) ) . An instance where the logging policy does not stabilize is , for example , when various actions have the same expected reward on certain subsets of the context space . Our approach in this paper is to use the variance-stabilization weights for the contextual bandit setting , just like Hadad et al . [ 2019 ] do the in the non-contextual bandit setting and Luedtke and van der Laan [ 2016 ] do in an iid setting with multiple optimal policies . Although Zhan et al . [ 2021 ] also tackle the contextual bandit setting , they do not use the variance-stabilization weights ; correspondingly their conditions and results apply in different , complementary settings . The key challenge we tackle in applying variance-stabilization weights in the contextual bandit setting is dealing with the need to use sequentially estimated weights .

Summary: The current paper concerns inference of quantities such as treatment effects or policy values from data that is collected from contextual bandit experiments, Motivated by applications in healthcare, revenue management or product recommendations where randomized experimentation is not often the norm. They isolate a key issue (variance stabilization), identified in previous works on analysis of adaptively collected data (see paper for references), and propose estimators for average causal effects of the  `doubly robust` form to overcome this issue. The main result is a central limit theorem for this estimator, allowing for frequentist inference (i.e. confidence intervals, p values) in the standard way. The estimator relies heavily on a sequence of conditional variance estimators, which  they construct using a judicious use of averaging and a base regression model for the outcome ( which may be misspecified even in the limit).