Before revision:  Based on the schemes, a new finite-time analysis of the Q-learning is given with a finite-time error bound. It offers novel intuitive insights on analysis of Q-learning mainly based on control theoretic frameworks.
After revision:  Based on the schemes, a new finite-time analysis of the Q-learning when a constant stepsize is used. Our analysis also sheds light on the overestimation phenomenon of Q-learning mainly based on control theoretic frameworks.