Before revision:  We show that contextual bandit algorithms are sensitive to the estimation method of the outcome model as well as the exploration method used, particularly in the presence of rich heterogeneity or complex outcome models, which can lead to difficult estimation problems along the path of learning. We propose new contextual bandit designs, combining parametric and non-parametric statistical estimation methods with causal inference methods in order to reduce the estimation biasthat results from adaptive treatment assignment .
After revision:  We develop parametric and non-parametric statistical estimation methods with causal inference methods in order to reduce the estimation biasthat results from adaptive treatment assignment .