The Adaptive Doubly Robust Estimator and a Paradox Concerning Logging Policy

1 Introduction . Adaptive experiments , including efficient treatment effect estimation ( van der Laan , 2008 ; Hahn et al. , 2011 ) , treatment regimes ( Zhang et al. , 2012a ; Zhao et al. , 2012 ) , and multi-armed bandit ( MAB ) problems ( Gittins , 1989 ; Lattimore & Szepesvári , 2020 ) , are widely accepted and used in real-world applications , for example , in social experiments ( Hahn et al. , 2011 ) , online advertisement ( Zhang et al. , 2012b ) , clinical trials ( Chow & Chang , 2011 ; Villar , 2018 ) , website optimization ( White , 2012 ) , and recommendation systems ( Li et al. , 2010 ) . In those various applications , there is a significant interest in off-line counterfactual inference using samples obtained from past trials generated by a logging policy ( the probability of choosing an action : Li et al. , 2010 , 2011 ) . Among several off-line evaluation criteria , this paper focuses on off-policy value estimation ( OPVE : Li et al. , 2015 ) . The goal of OPVE is to estimate the expected value of the weighted outcome , which includes average treatment effect ( ATE ) estimation as a special case ( Hirano et al. , 2003 ; Bang & Robins , 2005 ) . In OPVE , when the true logging policy is known , there are mainly three types of estimators : inverse probability weighting ( IPW : Horvitz & Thompson , 1952 ) , direct method ( DM ) , and augmented IPW ( AIPW Bang & Robins , 2005 ) estimators . IPW-type estimators refer to a sample average of the outcomes weighted by the true logging policy . DM-type estimators refer to a sample average of an estimated conditional outcome . AIPW-type estimators consist of two components : the IPW part and the DM part . More details are given in Section 3 . In addition , when the true logging policy is unknown , we call IPW-type and AIPW-type estimators that use the estimated logging policy EIPW-type and DR-type estimators , respectively . In particular , and the focus of this paper , DR-type estimators are crucial in OPVE , since in most applications of interest the true logging policy is either unknown , difficult , or costly to obtain , making IPW and AIPW-type estimators often infeasible . While existing OPVE methods typically assume that the samples are independent and identically distributed ( i.i.d . : Hirano et al. , 2003 ; Dudík et al. , 2011 ) , adaptive experiments usually allow logging policies to be updated based on past observations , under which the generated samples are non-i.i.d . ∗masahiro_kato @ cyberagent.co.jp 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In this case , theoretical results shown under i.i.d . assumptions , such as consistency and asymptotic normality , are not guaranteed . In particular , the asymptotic normality is critical , as we can obtain√ T -rate confidence intervals , where T is the sample size . For this problem , van der Laan ( 2008 ) proposed an asymptotically normal AIPW estimator under dependent samples , with several follow-up studies , including Luedtke & van der Laan ( 2016 , 2018 ) , Hadad et al . ( 2021 ) , Kato et al . ( 2020 ) , and Zhan et al . ( 2021 ) . However , a DR-type estimator for dependent samples has not been formally proposed , despite its importance . Readers might think that , in many experimental designs , the experiment can be manipulated by the experimenter , thus the information of the logging policy is always accessible . However , the logging policy is often not recorded . This is because it is costly to do so in adaptive experiments , since we need to preserve a policy for each sample . Consider the case of Thompson sampling , where we select the arm with the highest expected reward sampled from the posterior distribution ( Thompson , 1933 ) . In this case , if we want to know the logging policy , we need to calculate the probability that each arm is chosen from the posterior distribution . The calculation of this probability is not necessary for the bandit algorithm itself , but is an additional task for policy evaluation . In addition , to compute the logging policy , we not only need to record the information of the posterior distribution , but also need to calculate the probability by Monte Carlo simulation from the posterior distribution , if no analytical solution is available . This is a difficult task in terms of both computational volume and computational resource . Moreover , in adaptive experiments in industry ( e.g . AB tests ) , the number of logs is often in the order of several million . In such cases , storing the logging policy is also a difficult task . By using DR-type estimators , we can alleviate this cost . Further , as we report below as a paradox , it is even possible that DR-type estimators , that do not use the true logging policy , outperform estimators using the true logging policy , This paper proposes an asymptotically normal DR estimator for dependent samples . We call this DR estimator the adaptive DR ( ADR ) estimator . There are mainly two difficulties concerning this estimator . First , we can not use the central limit theorem ( CLT ) for i.i.d . or martingale difference sequences directly , as done in van der Laan ( 2008 ) , Hadad et al . ( 2021 ) , and Kato et al . ( 2020 ) . Second , when nuisance parameters are estimated with complicated models , the Donsker condition does not hold in general , which is required to show the asymptotic normality of semiparametric estimators , such as our proposed ADR estimator . In this paper , to solve these problems , we propose adaptive-fitting , inspired by the double/debiased machine learning for i.i.d . samples ( Chernozhukov et al. , 2018 ) . We also find that by using the ADR estimator , not only can OPVE be done without knowing the true logging policy , the ADR estimator paradoxically often outperforms the performance of AIPW estimators that use the true logging policy . We list our contributions as follows . ( I ) Asymptotic normality of the ADR estimator . We show the asymptotic normality of our proposed ADR estimator . To the best of our knowledge , a DR-type estimator for dependent samples obtained in adaptive experiments has not been formally proposed . ( II ) Adaptive fitting . While the asymptotic normalities of IPW and AIPW estimators are shown by the martingale CLT , we can not directly apply this technique to our proposed ADR estimator . This is because the martingale condition does not hold when the true logging policy is unknown . To solve this problem , we utilize a novel sample-splitting method called adaptive-fitting to show the asymptotic normality of the ADR estimator . We generalize this technique as a variant of samplesplitting ( Chernozhukov et al. , 2018 ) for semiparametric inference with dependent samples without the Donsker condition . Unlike existing sample-splitting methods , which assume that samples are i.i.d. , the proposed adaptive-fitting is applicable to non-i.i.d . samples . ( III ) Empirical paradox on using estimated logging policy . Through experimental studies , we investigate the empirical performance of our proposed ADR estimator . We find that the ADR estimator often exhibits improved performances over other estimators using the true logging policy , such as the AIPW estimator . Estimators requiring the true logging policy tend to empirically be unstable due to the instability of the nuisance parameter ; the instability being caused by the logging policy being near zero before convergence . Our finding implies that the ADR estimator mitigates this instability . We call this phenomenon a paradox concerning logging policy because using more information ( the true logging policy ) does not improve empirical performance . A similar paradox is known for IPW-type estimators for i.i.d . samples ( Hahn , 1998 ; Henmi & Eguchi , 2004 ) . However , we can not explain our paradox from a conventional semiparametric efficiency perspective ( Hahn , 1998 ; Hirano et al. , 2003 ) , as the asymptotic variances are the same between the ADR estimator ( with an estimated logging policy ) and the AIPW estimator ( with the true logging policy ) , unlike IPW-type estimators . Organization of this paper . In Section 2 , we formulate our problem . In Section 3 , we introduce existing OPVE estimators for dependent samples . In Section 4 , we propose the ADR estimator , which is our main contribution . We call the method used for the ADR estimator adaptive-fitting , and generalize it in Section 5 . In Section 6 , we report a paradox through simulation studies and explain the cause of this phenomenon . In Section 7 , we conduct experiments using benchmark datasets . 2 Problem setting . Suppose that there is a time series 1 , 2 , . . . , T , and denote the set as [ T ] = { 1 , . . . , T } . For t ∈ [ T ] , let At be an action in A = { 1 , 2 , . . . , K } , Xt be a covariate observed by a decision maker when choosing an action , and X be its space . Following the Neyman–Rubin causal model ( Rubin , 1974 ) , let a reward at period t be Yt = ∑K a=1 1 [ At = a ] Yt ( a ) , where Yt ( a ) : A → R is a potential ( random ) outcome . In summary , we have a dataset { ( Xt , At , Yt ) } T t=1 with the following data-generating process ( DGP ) : Step 1 : we observe a covariate Xt ∼ p ( x ) ; Step 2 : we choose At = a with probability pt ( a|x ) ; Step 3 : we observe an outcome Yt = ∑K a=1 1 [ At = a ] Yt ( a ) , where Yt ( a ) ∼ pa ( y|x ) , where p ( x ) denotes the density of the covariate Xt , pt ( a|x ) denotes the probability of choosing an action a conditional on a covariate x at period t , and pa ( y|x ) denotes the density of a reward Yt ( a ) conditional on a covariate x . We assume that p ( x ) and pa ( y|x ) are invariant across periods , and Xt and Yt ( a ) are drawn from these densities independently of the other period random variables ; that is , { ( Xt , Yt ( 1 ) , . . . , Yt ( K ) ) } Tt=1 is i.i.d. , but pt ( a|x ) can take different values across periods based on past observations . In this case , the samples { ( Xt , At , Yt ) } T t=1 are correlated over time , that is , the samples are not i.i.d . Let Ωt−1 = { Xt−1 , At−1 , Yt−1 , . . . , X1 , A1 , Y1 } be the history andMt−1 be a set of possible histories until the t-th period . The probability pt ( a|x ) is determined by a logging policy πt : A× X ×Mt−1 → ( 0 , 1 ) , such that ∑K a=1 πt ( a|x , Ωt−1 ) = 1 , which is a function of a covariate Xt , an action At , and history Ωt−1 . We also assume that πt is conditionally independent of Yt ( a ) to satisfy unconfoundedness ( Remark 2 ) . Remark 1 ( Stable unit treatment value assumption ( SUTVA ) ) . The DGP implies SUTVA ; that is , pa ( y|x ) is invariant to random assignments in other periods ( Angrist et al. , 1996 ) . Remark 2 ( Unconfoundedness ) . In this paper , unconfoundedness refers to independence between ( Yt ( 1 ) , . . . , Yt ( K ) ) and At , conditional on Xt and Ωt−1 , which is required for identification .

Summary: The authors propose and analyze an estimator for off policy value estimation (OPVE) in a Markovian setting where (X_t,A_t,Y_t(A_t)) ~ p(x)p_t(a|x)p(y_a|x) and the logging policy pi_t(a|x,Omega_{t-1}) is unknown. The estimator involves a familiar doubly robust moment function and a new form of cross fitting called adaptive fitting. The authors prove consistency, asymptotic normality, and semiparametric efficiency. Thorough simulations demonstrate low RMSE and correct coverage.