A Finite-Time Analysis of  Q-Learning with Neural Network Function Approximation

√ T ) convergence rate if the neural function approximator is sufficiently overparameterized , where T is the number of iterations . To our best knowledge , our result is the first finite-time analysis of neural Q-learning under non-i.i.d . data assumption . 1 INTRODUCTION . Q-learning has been shown to be one of the most important and effective learning strategies in Reinforcement Learning ( RL ) over the past decades ( Watkins & Dayan , 1992 ; Schmidhuber , 2015 ; Sutton & Barto , 2018 ) , where the agent takes an action based on the action-value function ( a.k.a. , Qvalue function ) at the current state . Recent advance in deep learning has also enabled the application of Q-learning algorithms to large-scale decision problems such as mastering Go ( Silver et al. , 2016 ; 2017 ) , robotic motion control ( Levine et al. , 2015 ; Kalashnikov et al. , 2018 ) and autonomous driving ( Shalev-Shwartz et al. , 2016 ; Schwarting et al. , 2018 ) . In particular , the seminal work by Mnih et al . ( 2015 ) introduced the Deep Q-Network ( DQN ) to approximate the action-value function and achieved a superior performance versus a human expert in playing Atari games , which triggers a line of research on deep reinforcement learning such as Double Deep Q-Learning ( Van Hasselt et al. , 2016 ) and Dueling DQN ( Wang et al. , 2016 ) . Apart from its widespread empirical success in numerous applications , the convergence of Qlearning and temporal difference ( TD ) learning algorithms has also been extensively studied in the literature ( Jaakkola et al. , 1994 ; Baird , 1995 ; Tsitsiklis & Van Roy , 1997 ; Perkins & Pendrith , 2002 ; Melo et al. , 2008 ; Mehta & Meyn , 2009 ; Liu et al. , 2015 ; Bhandari et al. , 2018 ; Lakshminarayanan & Szepesvari , 2018 ; Zou et al. , 2019b ) . However , the convergence guarantee of deep Q-learning algorithms remains a largely open problem . The only exceptions are Yang et al . ( 2019 ) which studied the fitted Q-iteration ( FQI ) algorithm ( Riedmiller , 2005 ; Munos & Szepesvári , 2008 ) with action-value function approximation based on a sparse ReLU network , and Cai et al . ( 2019a ) which studied the global convergence of Q-learning algorithm with an i.i.d . observation model and action-value function approximation based on a two-layer neural network . The main limitation of the aforementioned work is the unrealistic assumption that all the data used in the Q-learning algorithm are sampled i.i.d . from a fixed stationary distribution , which fails to capture the practical setting of neural Q-learning . In this paper , in order to bridge the gap between the empirical success of neural Q-learning and the theory of conventional Q-learning ( i.e. , tabular Q-learning , and Q-learning with linear function approximation ) , we study the non-asymptotic convergence of a neural Q-learning algorithm under non-i.i.d . observations . In particular , we use a deep neural network with the ReLU activation function to approximate the action-value function . In each iteration of the neural Q-learning algorithm , it updates the network weight parameters using the temporal difference ( TD ) error and the gradient of the neural network function . Our work extends existing finite-time analyses for TD learning ( Bhandari et al. , 2018 ) and Q-learning ( Zou et al. , 2019b ) , from linear function approximation to deep neural network based function approximation . Compared with the very recent theoretical work for neural Q-learning ( Yang et al. , 2019 ; Cai et al. , 2019a ) , our analysis relaxes the non-realistic i.i.d . data assumption and applies to neural network approximation with arbitrary number of layers . Our main contributions are summarized as follows • We establish the first finite-time analysis of Q-learning with deep neural network function approximation when the data are generated from a Markov decision process ( MDP ) . We show that , when the network is sufficiently wide , neural Q-learning converges to the optimal action-value function up to the approximation error of the neural network function class . • We establish an O ( 1/ √ T ) convergence rate of neural Q-learning to the optimal Q-value function up to the approximation error , where T is the number of iterations . This convergence rate matches the one for TD-learning with linear function approximation and constant stepsize ( Bhandari et al. , 2018 ) . Although we study a more challenging setting where the data are non-i.i.d . and the neural network approximator has multiple layers , our convergence rate also matches the O ( 1/ √ T ) rate proved in Cai et al . ( 2019a ) with i.i.d . data and a two-layer neural network approximator . To sum up , we present a comprehensive comparison between our work and the most relevant work in terms of their respective settings and convergence rates in Table 1 . Notation We denote [ n ] = { 1 , . . . , n } for n ∈ N+ . ‖x‖2 is the Euclidean norm of a vector x ∈ Rd . For a matrix W ∈ Rm×n , we denote by ‖W‖2 and ‖W‖F its operator norm and Frobenius norm respectively . We denote by vec ( W ) the vectorization of W , which converts W into a column vector . For a semi-definite matrix Σ ∈ Rd×d and a vector x ∈ Rd , ‖x‖Σ = √ x > Σx denotes the Mahalanobis norm . We reserve the notations { Ci } i=0,1 , ... to represent universal positive constants that are independent of problem parameters . The specific value of { Ci } i=1,2 , ... can be different line by line . We write an = O ( bn ) if an ≤ Cbn for some constant C > 0 and an = Õ ( bn ) if an = O ( bn ) up to some logarithmic terms of bn . 2 RELATED WORK . Due to the huge volume of work in the literature for TD learning and Q-learning algorithms , we only review the most relevant work here . Asymptotic analysis The asymptotic convergence of TD learning and Q-learning algorithms has been well established in the literature ( Jaakkola et al. , 1994 ; Tsitsiklis & Van Roy , 1997 ; Konda & Tsitsiklis , 2000 ; Borkar & Meyn , 2000 ; Ormoneit & Sen , 2002 ; Melo et al. , 2008 ; Devraj & Meyn , 2017 ) . In particular , Tsitsiklis & Van Roy ( 1997 ) specified the precise conditions for TD learning with linear function approximation to converge and gave counterexamples that diverge . Melo et al . ( 2008 ) proved the asymptotic convergence of Q-learning with linear function approximation from standard ODE analysis , and identified a critic condition on the relationship between the learning policy and the greedy policy that ensures the almost sure convergence . Finite-time analysis The finite-time analysis of the convergence rate for Q-learning algorithms has been largely unexplored until recently . In specific , Dalal et al . ( 2018 ) ; Lakshminarayanan & Szepesvari ( 2018 ) studied the convergence of TD ( 0 ) algorithm with linear function approximation under i.i.d . data assumptions and constant step sizes . Concurrently , a seminal work by Bhandari et al . ( 2018 ) provided a unified framework of analysis for TD learning under both i.i.d . and Markovian noise assumptions with an extra projection step . The analysis has been extended by Zou et al . ( 2019b ) to SARSA and Q-learning algorithms with linear function approximation . More recently , Srikant & Ying ( 2019 ) established the finite-time convergence for TD learning algorithms with linear function approximation and a constant step-size without the extra projection step under non-i.i.d . data assumptions through carefully choosing the Lyapunov function for the associated ordinary differential equation of TD update . A similar analysis was also extended to Q-learning with linear function approximation ( Chen et al. , 2019 ) . Hu & Syed ( 2019 ) further provided a unified analysis for a class of TD learning algorithms using Markov jump linear system . Neural function approximation Despite the empirical success of DQN , the theoretical convergence of Q-learning with deep neural network approximation is still missing in the literature . Following the recent advances in the theory of deep learning for overparameterized networks ( Jacot et al. , 2018 ; Chizat & Bach , 2018 ; Du et al. , 2019b ; a ; Allen-Zhu et al. , 2019b ; a ; Zou et al. , 2019a ; Arora et al. , 2019 ; Cao & Gu , 2019a ; Zou & Gu , 2019 ; Cai et al. , 2019b ) , two recent work by Yang et al . ( 2019 ) and Cai et al . ( 2019a ) proved the convergence rates of fitted Q-iteration and Q-learning with a sparse multi-layer ReLU network and two-layer neural network approximation respectively , under i.i.d . observations . 3 PRELIMINARIES . A discrete-time Markov Decision Process ( MDP ) is denoted by a tupleM = ( S , A , P , r , γ ) . S and A are the sets of all states and actions respectively . P : S × A → P ( S ) is the transition kernel such that P ( s′|s , a ) gives the probability of transiting to state s′ after taking action a at state s. r : S ×A → [ −1 , 1 ] is a deterministic reward function . γ ∈ ( 0 , 1 ) is the discounted factor . A policy π : S → P ( A ) is a function mapping a state s ∈ S to a probability distribution π ( ·|s ) over the action space . Let st and at denote the state and action at time step t. Then the transition kernel P and the policy π determine a Markov chain { st } t=0,1 , ... For any fixed policy π , its associated value function V π : S → R is defined as the expected total discounted reward : V π ( s ) = E [ ∑∞ t=0 γ tr ( st , at ) |s0 = s ] , ∀s ∈ S. The corresponding action-value function Qπ : S ×A → R is defined as Qπ ( s , a ) = E [ ∑∞ t=0 γ tr ( st , at ) |s0 = s , a0 = a ] = r ( s , a ) + γ ∫ S V π ( s′ ) P ( s′|s , a ) ds′ , for all s ∈ S , a ∈ A . The optimal action-value function Q∗ is defined as Q∗ ( s , a ) = supπ Qπ ( s , a ) for all ( s , a ) ∈ S × A . Based on Q∗ , the optimal policy π∗ can be derived by following the greedy algorithm such that π∗ ( a|s ) = 1 if Q ( s , a ) = maxb∈AQ∗ ( s , b ) and π∗ ( a|s ) = 0 otherwise . We define the optimal Bellman operator T as follows T Q ( s , a ) = r ( s , a ) + γ · E [ maxb∈AQ ( s ′ , b ) |s′ ∼ P ( ·|s , a ) ] . ( 3.1 ) It is worth noting that the optimal Bellman operator T is γ-contractive in the sup-norm and Q∗ is the unique fixed point of T ( Bertsekas et al. , 1995 ) .

Summary: This paper studies the convergence of Q-Learning when a wide multi-layer network in the Neural Tangent Kernel (NTK) regime is used as the function approximator. Concretely, it shows that Q-learning converges with rate O(1/T) with data sampled from a single trajectory (non-i.i.d.) of an infinite-horizon MDP + a certain (stationary) exploration policy.