Before revision:  It offers novel and intuitive insights on Q-learning mainly based on control theoretic frameworks. For instance, the proposed analysis explains the overestimation phenomenon in Q-learning due to the maximization bias . Based on the control system theoretic argument and some nice structures of Q-learning , a new finite-time analysis of the Q-learning is given with a novel error bound .
After revision:  It offers novel and intuitive insights on Q-learning estimation error is over- and underestimated by trajectories of two dynamical systems . Based on the control system theoretic argument and some nice structures of Q-learning , a new finite-time analysis of the Q-learning is given with a novel error bound .