Before revision: This paper develops a novel  framework to analyze the  convergence of Q-learning algorithm by using its connections to dynamical systems.
After revision: This paper develops a novel framework to analyze the non-asymptotic convergence of Q-learning algorithm by using its connections to dynamical systems.