The Adaptive Doubly Robust Estimator and a Paradox Concerning Logging Policy

1 Introduction . Adaptive experiments , including efficient treatment effect estimation ( van der Laan , 2008 ; Hahn et al. , 2011 ) , treatment regimes ( Zhang et al. , 2012a ; Zhao et al. , 2012 ) , and multi-armed bandit ( MAB ) problems ( Gittins , 1989 ; Lattimore & Szepesvári , 2020 ) , are widely accepted and used in real-world applications , for example , in social experiments ( Hahn et al. , 2011 ) , online advertisement ( Zhang et al. , 2012b ) , clinical trials ( Chow & Chang , 2011 ; Villar , 2018 ) , website optimization ( White , 2012 ) , and recommendation systems ( Li et al. , 2010 ) . In those various applications , there is a significant interest in off-line counterfactual inference using samples obtained from past trials generated by a logging policy ( the probability of choosing an action : Li et al. , 2010 , 2011 ) . Among several off-line evaluation criteria , this paper focuses on off-policy value estimation ( OPVE : Li et al. , 2015 ) . The goal of OPVE is to estimate the expected value of the weighted outcome , which includes average treatment effect ( ATE ) estimation as a special case ( Hirano et al. , 2003 ; Bang & Robins , 2005 ) . In OPVE , when the true logging policy is known , there are mainly three types of estimators : inverse probability weighting ( IPW : Horvitz & Thompson , 1952 ) , direct method ( DM ) , and augmented IPW ( AIPW Bang & Robins , 2005 ) estimators . IPW-type estimators refer to a sample average of the outcomes weighted by the true logging policy . DM-type estimators refer to a sample average of an estimated conditional outcome . AIPW-type estimators consist of two components : the IPW part and the DM part . More details are given in Section 3 . In addition , when the true logging policy is unknown , we call IPW-type and AIPW-type estimators that use the estimated logging policy EIPW-type and DR-type estimators , respectively . In particular , and the focus of this paper , DR-type estimators are crucial in OPVE , since in most applications of interest the true logging policy is either unknown , difficult , or costly to obtain , making IPW and AIPW-type estimators often infeasible . While existing OPVE methods typically assume that the samples are independent and identically distributed ( i.i.d . : Hirano et al. , 2003 ; Dudík et al. , 2011 ) , adaptive experiments usually allow logging policies to be updated based on past observations , under which the generated samples are non-i.i.d . ∗masahiro_kato @ cyberagent.co.jp 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In this case , theoretical results shown under i.i.d . assumptions , such as consistency and asymptotic normality , are not guaranteed . In particular , the asymptotic normality is critical , as we can obtain√ T -rate confidence intervals , where T is the sample size . For this problem , van der Laan ( 2008 ) proposed an asymptotically normal AIPW estimator under dependent samples , with several follow-up studies , including Luedtke & van der Laan ( 2016 , 2018 ) , Hadad et al . ( 2021 ) , Kato et al . ( 2020 ) , and Zhan et al . ( 2021 ) . However , a DR-type estimator for dependent samples has not been formally proposed , despite its importance . Readers might think that , in many experimental designs , the experiment can be manipulated by the experimenter , thus the information of the logging policy is always accessible . However , the logging policy is often not recorded . This is because it is costly to do so in adaptive experiments , since we need to preserve a policy for each sample . Consider the case of Thompson sampling , where we select the arm with the highest expected reward sampled from the posterior distribution ( Thompson , 1933 ) . In this case , if we want to know the logging policy , we need to calculate the probability that each arm is chosen from the posterior distribution . The calculation of this probability is not necessary for the bandit algorithm itself , but is an additional task for policy evaluation . In addition , to compute the logging policy , we not only need to record the information of the posterior distribution , but also need to calculate the probability by Monte Carlo simulation from the posterior distribution , if no analytical solution is available . This is a difficult task in terms of both computational volume and computational resource . Moreover , in adaptive experiments in industry ( e.g . AB tests ) , the number of logs is often in the order of several million . In such cases , storing the logging policy is also a difficult task . By using DR-type estimators , we can alleviate this cost . Further , as we report below as a paradox , it is even possible that DR-type estimators , that do not use the true logging policy , outperform estimators using the true logging policy , This paper proposes an asymptotically normal DR estimator for dependent samples . We call this DR estimator the adaptive DR ( ADR ) estimator . There are mainly two difficulties concerning this estimator . First , we can not use the central limit theorem ( CLT ) for i.i.d . or martingale difference sequences directly , as done in van der Laan ( 2008 ) , Hadad et al . ( 2021 ) , and Kato et al . ( 2020 ) . Second , when nuisance parameters are estimated with complicated models , the Donsker condition does not hold in general , which is required to show the asymptotic normality of semiparametric estimators , such as our proposed ADR estimator . In this paper , to solve these problems , we propose adaptive-fitting , inspired by the double/debiased machine learning for i.i.d . samples ( Chernozhukov et al. , 2018 ) . We also find that by using the ADR estimator , not only can OPVE be done without knowing the true logging policy , the ADR estimator paradoxically often outperforms the performance of AIPW estimators that use the true logging policy . We list our contributions as follows . ( I ) Asymptotic normality of the ADR estimator . We show the asymptotic normality of our proposed ADR estimator . To the best of our knowledge , a DR-type estimator for dependent samples obtained in adaptive experiments has not been formally proposed . ( II ) Adaptive fitting . While the asymptotic normalities of IPW and AIPW estimators are shown by the martingale CLT , we can not directly apply this technique to our proposed ADR estimator . This is because the martingale condition does not hold when the true logging policy is unknown . To solve this problem , we utilize a novel sample-splitting method called adaptive-fitting to show the asymptotic normality of the ADR estimator . We generalize this technique as a variant of samplesplitting ( Chernozhukov et al. , 2018 ) for semiparametric inference with dependent samples without the Donsker condition . Unlike existing sample-splitting methods , which assume that samples are i.i.d. , the proposed adaptive-fitting is applicable to non-i.i.d . samples . ( III ) Empirical paradox on using estimated logging policy . Through experimental studies , we investigate the empirical performance of our proposed ADR estimator . We find that the ADR estimator often exhibits improved performances over other estimators using the true logging policy , such as the AIPW estimator . Estimators requiring the true logging policy tend to empirically be unstable due to the instability of the nuisance parameter ; the instability being caused by the logging policy being near zero before convergence . Our finding implies that the ADR estimator mitigates this instability . We call this phenomenon a paradox concerning logging policy because using more information ( the true logging policy ) does not improve empirical performance . A similar paradox is known for IPW-type estimators for i.i.d . samples ( Hahn , 1998 ; Henmi & Eguchi , 2004 ) . However , we can not explain our paradox from a conventional semiparametric efficiency perspective ( Hahn , 1998 ; Hirano et al. , 2003 ) , as the asymptotic variances are the same between the ADR estimator ( with an estimated logging policy ) and the AIPW estimator ( with the true logging policy ) , unlike IPW-type estimators . Organization of this paper . In Section 2 , we formulate our problem . In Section 3 , we introduce existing OPVE estimators for dependent samples . In Section 4 , we propose the ADR estimator , which is our main contribution . We call the method used for the ADR estimator adaptive-fitting , and generalize it in Section 5 . In Section 6 , we report a paradox through simulation studies and explain the cause of this phenomenon . In Section 7 , we conduct experiments using benchmark datasets . 2 Problem setting . Suppose that there is a time series 1 , 2 , . . . , T , and denote the set as [ T ] = { 1 , . . . , T } . For t ∈ [ T ] , let At be an action in A = { 1 , 2 , . . . , K } , Xt be a covariate observed by a decision maker when choosing an action , and X be its space . Following the Neyman–Rubin causal model ( Rubin , 1974 ) , let a reward at period t be Yt = ∑K a=1 1 [ At = a ] Yt ( a ) , where Yt ( a ) : A → R is a potential ( random ) outcome . In summary , we have a dataset { ( Xt , At , Yt ) } T t=1 with the following data-generating process ( DGP ) : Step 1 : we observe a covariate Xt ∼ p ( x ) ; Step 2 : we choose At = a with probability pt ( a|x ) ; Step 3 : we observe an outcome Yt = ∑K a=1 1 [ At = a ] Yt ( a ) , where Yt ( a ) ∼ pa ( y|x ) , where p ( x ) denotes the density of the covariate Xt , pt ( a|x ) denotes the probability of choosing an action a conditional on a covariate x at period t , and pa ( y|x ) denotes the density of a reward Yt ( a ) conditional on a covariate x . We assume that p ( x ) and pa ( y|x ) are invariant across periods , and Xt and Yt ( a ) are drawn from these densities independently of the other period random variables ; that is , { ( Xt , Yt ( 1 ) , . . . , Yt ( K ) ) } Tt=1 is i.i.d. , but pt ( a|x ) can take different values across periods based on past observations . In this case , the samples { ( Xt , At , Yt ) } T t=1 are correlated over time , that is , the samples are not i.i.d . Let Ωt−1 = { Xt−1 , At−1 , Yt−1 , . . . , X1 , A1 , Y1 } be the history andMt−1 be a set of possible histories until the t-th period . The probability pt ( a|x ) is determined by a logging policy πt : A× X ×Mt−1 → ( 0 , 1 ) , such that ∑K a=1 πt ( a|x , Ωt−1 ) = 1 , which is a function of a covariate Xt , an action At , and history Ωt−1 . We also assume that πt is conditionally independent of Yt ( a ) to satisfy unconfoundedness ( Remark 2 ) . Remark 1 ( Stable unit treatment value assumption ( SUTVA ) ) . The DGP implies SUTVA ; that is , pa ( y|x ) is invariant to random assignments in other periods ( Angrist et al. , 1996 ) . Remark 2 ( Unconfoundedness ) . In this paper , unconfoundedness refers to independence between ( Yt ( 1 ) , . . . , Yt ( K ) ) and At , conditional on Xt and Ωt−1 , which is required for identification .

Summary: The paper studies off-policy evaluation from data generated from unknown and adaptive logging policies (logging policy can be updated based on past observations). The paper modified the doubly robust estimator in the non-adaptive and known-logging-policy setting to this setting by using an estimated logging policy and an estimated reward model. The estimation models used at each time step are built only on data collected before. They show the asymptotic normality of this estimator under some assumptions and make the step-wise construction and how it achieves asymptotic normality as a general approach. They empirically evaluated the proposed estimator on some simulation dataset and  discussed why the proposed estimator enjoys smaller mean squared error empirically compared to estimators with known logging policy information, even when they all have asymptotic normality.