Before revision: We theoretically and experimentally compare estimators for off-policy evaluation (OPE) using dependent samples obtained via multi-armed bandit (MAB) algorithms. The goal of OPE is to evaluate a new policy using historical data. Because the MAB algorithms sequentially update the policy based on past observations, the generated samples are not independent and identically distributed. To conduct OPE from dependent samples , we need to use some techniques for constructing the estimator with asymptotic normality. In particular, we focus on a doubly robust (DR ) estimator , which consists of an inverse probability weighting (IPW) component and an estimatorof the conditionally expected outcome. We first summarize existing and new theoretical results for such OPE estimators .
After revision: A doubly robust (DR) estimator is crucial in causal inference, which consists of two nuisance parameters: the conditional mean outcome and logging policy (probability of choosing an action). This paper provides a DR estimator for dependent samples obtained in adaptive experiments and introduces two related topics. First, we propose adaptive-fitting as a variant of sample-splitting for showing an asymptotically normal semiparametric estimator from dependent samples without non-Donsker nuisance estimators. Second, we report an empirical paradox that a DR estimator shows better performances than other estimators using the true logging policy instead of its estimator .