Statistical Inference with M-Estimators on Adaptively Collected Data

1 Introduction . Due to the need for interventions that are personalized to users , ( contextual ) bandit algorithms are increasingly used to address sequential decision making problems in health-care [ Yom-Tov et al. , 2017 , Liao et al. , 2020 ] , online education [ Liu et al. , 2014 , Shaikh et al. , 2019 ] , and public policy [ Kasy and Sautmann , 2021 , Caria et al. , 2020 ] . Contextual bandits personalize , that is , minimize regret , by learning to choose the best intervention in each context , i.e. , the action that leads to the greatest expected reward . Besides the goal of regret minimization , another critical goal in these real-world problems is to be able to use the resulting data collected by bandit algorithms to advance scientific knowledge [ Liu et al. , 2014 , Erraqabi et al. , 2017 ] . By scientific knowledge , we mean information gained by using the data to conduct a variety of statistical analyses , including confidence interval construction and hypothesis testing . While regret minimization is a within-experiment learning objective , gaining scientific knowledge from the resulting adaptively collected data is a between-experiment learning objective , which ultimately helps with regret minimization between deployments of bandit algorithms . Note that the data collected by bandit algorithms are adaptively collected because previously observed contexts , actions , and rewards are used to inform what actions to select in future timesteps . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . There are a variety of between-experiment learning questions encountered in real-life applications of bandit algorithms . For example , in real-life sequential decision-making problems there are often a number of additional scientifically interesting outcomes besides the reward that are collected during the experiment . In the online advertising setting , the reward might be whether an ad is clicked on , but one may be interested in the outcome of amount of money spent or the subsequent time spent on the advertiser ’ s website . If it was found that an ad had high click-through rate , but low amounts of money was spent after clicking on the ad , one may redesign the reward used in the next bandit experiment . One type of statistical analysis would be to construct confidence intervals for the relative effect of the actions on multiple outcomes ( in addition to the reward ) conditional on the context . Furthermore , due to engineering and practical limitations , some of the variables that might be useful as context are often not accessible to the bandit algorithm online . If after-study analyses find some such contextual variables to have sufficiently strong influence on the relative usefulness of an action , this might lead investigators to ensure these variables are accessible to the bandit algorithm in the next experiment . As discussed above , we can gain scientific knowledge from data collected with ( contextual ) bandit algorithms by constructing confidence intervals and performing hypothesis tests for unknown quantities such as the expected outcome for different actions in various contexts . Unfortunately , standard statistical methods developed for i.i.d . data fail to provide valid inference when applied to data collected with common bandit algorithms . For example , assuming the sample mean of rewards for an arm is approximately normal can lead to unreliable confidence intervals and inflated type-1 error ; see Section 3.1 for an illustration . Recently statistical inference methods have been developed for data collected using bandit algorithms [ Hadad et al. , 2019 , Deshpande et al. , 2018 , Zhang et al. , 2020 ] ; however , these methods are limited to inference for parameters of simple models . There is a lack of general statistical inference methods for data collected with ( contextual ) bandit algorithms in more complex data-analytic settings , including parameters in non-linear models for outcomes ; for example , there are currently no methods for constructing valid confidence intervals for the parameters of a logistic regression model for binary outcomes or for constructing confidence intervals based on robust estimators like minimizers of the Huber loss function . In this work we show that a wide variety of estimators which are frequently used both in science and industry on i.i.d . data , namely , M-estimators [ Van der Vaart , 2000 ] , can be used to conduct valid inference on data collected with ( contextual ) bandit algorithms when adjusted with particular adaptive weights , i.e. , weights that are a function of previously collected data . Different forms of adaptive weights are used by existing methods for simple models [ Deshpande et al. , 2018 , Hadad et al. , 2019 , Zhang et al. , 2020 ] . Our work is a step towards developing a general framework for statistical inference on data collected with adaptive algorithms , including ( contextual ) bandit algorithms . 2 Problem Formulation . We assume that the data we have after running a contextual bandit algorithm is comprised of contexts { Xt } Tt=1 , actions { At } Tt=1 , and primary outcomes { Yt } Tt=1 . T is deterministic and known . We assume that rewards are a deterministic function of the primary outcomes , i.e. , Rt = f ( Yt ) for some known function f . We are interested in constructing confidence regions for the parameters of the conditional distribution of Yt given ( Xt , At ) . Below we consider T ! 1 in order to derive the asymptotic distributions of estimators and construct asymptotically valid confidence intervals . We allow the action space A to be finite or infinite . We use potential outcome notation [ Imbens and Rubin , 2015 ] and let { Yt ( a ) : a 2 A } denote the potential outcomes of the primary outcome and let Yt : = Yt ( At ) be the observed outcome . We assume a stochastic contextual bandit environment in which { Xt , Yt ( a ) : a 2 A } i.i.d . ⇠ P 2 P for t 2 [ 1 : T ] ; the contextual bandit environment distribution P is in a space of possible environment distributions P. We define the history Ht : = { Xt0 , At0 , Yt0 } tt0=1 for t 1 and H0 : = ; . Actions At 2 A are selected according to policies ⇡ : = { ⇡t } t 1 , which define action selection probabilities ⇡t ( At , Xt , Ht 1 ) : = P ( At|Ht 1 , Xt ) . Even though the potential outcomes are i.i.d. , the observed data { Xt , At , Yt } Tt=1 are not because the actions are selected using policies ⇡t which are a function of past data , Ht 1 . Non-independence of observations is a key property of adaptively collected data . We are interested in constructing confidence regions for some unknown ✓⇤ ( P ) 2 ⇥ ⇢ Rd , which is a parameter of the conditional distribution of Yt given ( Xt , At ) . This work focuses on the setting in which we have a well-specified model for Yt . Specifically , we assume that ✓⇤ ( P ) is a conditionally maximizing value of criterion m✓ , i.e. , for all P 2 P , ✓⇤ ( P ) 2 argmax ✓2⇥ EP [ m✓ ( Yt , Xt , At ) |Xt , At ] w.p . 1 . ( 1 ) Note that ✓⇤ ( P ) does not depend on ( Xt , At ) and it is an implicit modelling assumption that such a ✓⇤ ( P ) exists for a given m✓ . Note that this formulation includes semi-parametric models , e.g. , the model could constrain the conditional mean of Yt to be linear in some function of the actions and context , but allow the residuals to follow any mean-zero distribution , including ones that depend on the actions and/or contexts . To estimate ✓⇤ ( P ) , we build on M-estimation [ Huber , 1992 ] , which classically selects the estimator ✓̂ to be the ✓ 2 ⇥ that maximizes the empirical analogue of Equation ( 1 ) : ✓̂T : = argmax ✓2⇥ 1 T TX t=1 m✓ ( Yt , Xt , At ) . ( 2 ) For example , in a classical linear regression setting with |A| < 1 actions , a natural choice for m✓ is the negative of the squared loss function , m✓ ( Yt , Xt , At ) = ( Yt X > t ✓At ) 2 . When Yt is binary , a natural choice is instead the negative log-likelihood function for a logistic regression model , i.e. , m✓ ( Yt , Xt , At ) = [ YtX > t ✓At log ( 1 + exp ( X > t ✓At ) ) ] . More generally , m✓ is commonly chosen to be a log-likelihood function or the negative of a robust loss function such as the Huber loss . If the data , { Xt , At , Yt } Tt=1 , were independent across time , classical approaches could be used to prove the consistency and asymptotic normality of M-estimators [ Van der Vaart , 2000 ] . However , on data collected with bandit algorithms , standard M-estimators like the ordinary least-squares estimator fail to provide valid confidence intervals [ Hadad et al. , 2019 , Deshpande et al. , 2018 , Zhang et al. , 2020 ] . In this work , we show that M-estimators can still be used to provide valid statistical inference on adaptively collected data when adjusted with well-chosen adaptive weights . 3 Adaptively Weighted M-Estimators . We consider a weighted M-estimating criteria with adaptive weights Wt 2 ( Ht 1 , Xt , At ) given by Wt = q ⇡stat ( At , Xt ) ⇡t ( At , Xt , Ht 1 ) . Here { ⇡stat } t 1 are pre-specified stabilizing policies that do not depend on data { Yt , Xt , At } t 1 . A default choice for the stabilizing policy when the action space is of size |A| < 1 is just ⇡stat ( a , x ) = 1/|A| for all x , a , and t ; we discuss considerations for the choice of { ⇡stat } T t=1 in Section 3.3 . We call these weights square-root importance weights because they are the square-root of the standard importance weights [ Hammersley , 2013 , Wang et al. , 2017 ] . Our proposed estimator for ✓⇤ ( P ) , ✓̂T , is the maximizer of a weighted version of the M-estimation criterion of Equation ( 2 ) : ✓̂T : = argmax ✓2⇥ 1 T TX t=1 Wtm✓ ( Yt , Xt , At ) = : argmax ✓2⇥ MT ( ✓ ) . Note that MT ( ✓ ) defined above depends on both the data { Xt , At , Yt } Tt=1 and weights { Wt } Tt=1 . We provide asymptotically valid confidence regions for ✓⇤ ( P ) by deriving the asymptotic distribution of ✓̂T as T ! 1 and by proving that the convergence in distribution is uniform over P 2 P. Such convergence allows us to construct a uniformly asymptotically valid 1 ↵ level confidence region , CT ( ↵ ) , for ✓⇤ ( P ) , which is a confidence region that satisfies lim inf T ! 1 inf P2P PP , ⇡ ( ✓⇤ ( P ) 2 CT ( ↵ ) ) 1 ↵ . ( 3 ) If CT ( ↵ ) were not uniformly valid , then there would exist an ✏ > 0 such that for every sample size T , CT ( ↵ ) ’ s coverage would be below 1 ↵ ✏ for some worst-case PT 2 P. Confidence regions which are asymptotically valid , but not uniformly asymptotically valid , fail to be reliable in practice [ Leeb and Pötscher , 2005 , Romano et al. , 2012 ] . Note that on i.i.d . data it is generally straightforward to show that estimators that converge in distribution do so uniformly ; however , as discussed in Zhang et al . [ 2020 ] and Appendix D , this is not the case on data collected with bandit algorithms . To construct uniformly valid confidence regions for ✓⇤ ( P ) we prove that ✓̂T is uniformly asymptotically normal in the following sense : ⌃T ( P ) 1/2M̈T ( ✓̂T ) p T ( ✓̂T ✓ ⇤ ( P ) ) D ! N ( 0 , Id ) uniformly over P 2 P , ( 4 ) where M̈T ( ✓ ) : = @ 2 @ 2✓MT ( ✓ ) and ⌃T ( P ) : = 1 T PT t=1 EP , ⇡stat ⇥ ṁ✓⇤ ( P ) ( Yt , Xt , At ) ⌦2 ⇤ . We define ṁ✓ : = @ @ ✓m✓ . Similarly we define respectively m̈✓ and ... m✓ as the second and third partial derivatives of m✓ with respect to ✓ . For any vector z we define z⌦2 : = zz > .

Summary: The authors analyze square root importance sampling weighted M-estimators over parametric classes for adaptively collected data.   This work is motivated by the fact that the usual unweigthed M-estimators that one would use for i.i.d. data fail to be asymptotically normally distributed under adaptively collected data.  Under appropriate conditions, the proposed weighting scheme makes the resulting M-estimator asymptotically normal, uniformly over the parameter space.  The authors demonstrate their claims through simulations.