Q-learning with UCB Exploration is Sample Efficient for Infinite-Horizon MDP
A fundamental question in reinforcement learning is whether model-free algorithms are sample efficient.Recently, Jin et al. proposed a Q-learning algorithm with UCB exploration policy, and proved it has nearly optimal regret bound for finite-horizon episodic MDP.In this paper, we adapt Q-learning with UCB-exploration bonus to infinite-horizon MDP with discounted rewards accessing a generative model.We show that the of our algorithm is bounded by.This improves the previously best known result of in this setting achieved by delayed Q-learning,, and matches the lower bound in terms of as well as and up to logarithmic factors.
We adapt Q-learning with UCB-exploration bonus to infinite-horizon MDP with discounted rewards without accessing a generative model, and improves the previously best known result.This paper considered a Q-learning algorithm with UCB exploration policy for infinite-horizon MDP.