Before revision: This paper develops a novel  framework to analyze the  convergence of Q-learning algorithm by using its connections to dynamical systems. We prove that asynchronous Q-learning with a constant step-size can be naturally formulated as  discrete-time stochastic switched linear systems .
After revision: This paper develops a novel framework to analyze the convergence of Q-learning . We show that the dynamics of asynchronous Q-learning with a constant step-size can be naturally formulated as discrete-time stochastic switched linear systems .