A Theoretical Analysis of  Deep Q-Learning

Despite the great empirical success of deep reinforcement learning , its theoretical foundation is less well understood . In this work , we make the first attempt to theoretically understand the deep Q-network ( DQN ) algorithm ( Mnih et al. , 2015 ) from both algorithmic and statistical perspectives . In specific , we focus on the fitted Q iteration ( FQI ) algorithm with deep neural networks , which is a slight simplification of DQN that captures the tricks of experience replay and target network used in DQN . Under mild assumptions , we establish the algorithmic and statistical rates of convergence for the action-value functions of the iterative policy sequence obtained by FQI . In particular , the statistical error characterizes the bias and variance that arise from approximating the action-value function using deep neural network , while the algorithmic error converges to zero at a geometric rate . As a byproduct , our analysis provides justifications for the techniques of experience replay and target network , which are crucial to the empirical success of DQN . Furthermore , as a simple extension of DQN , we propose the Minimax-DQN algorithm for zero-sum Markov game with two players , which is deferred to the appendix due to space limitations . 1 INTRODUCTION . Reinforcement learning ( RL ) attacks the multi-stage decision-making problems by interacting with the environment and learning from the experiences . With the breakthrough in deep learning , deep reinforcement learning ( DRL ) demonstrates tremendous success in solving highly challenging problems , such as the game of Go ( Silver et al. , 2016 ; 2017 ) , robotics ( Kober & Peters , 2012 ) , and dialogue systems ( Chen et al. , 2017 ) . In DRL , the value or policy functions are often represented as deep neural networks and the related deep learning techniques can be readily applied . For example , deep Q-network ( DQN ) ( Mnih et al. , 2015 ) , asynchronous advantage actor-critic ( A3C ) and ( Mnih et al. , 2016 ) demonstrate superhuman performance in various applications and become standard algorithms for artificial intelligence . Despite its great empirical success , there exists a gap between the theory and practice of DRL . In particular , most existing theoretical work on reinforcement learning focuses on the tabular case where the state and action spaces are finite , or the case where the value function is linear . Under these restrictive settings , the algorithmic and statistical perspectives of reinforcement learning are well-understood via the tools developed for convex optimization and linear regression . However , in presence of nonlinear function approximators such as deep neural network , the theoretical analysis of reinforcement learning becomes intractable as it involves solving a highly nonconvex statistical optimization problem . To bridge such a gap in DRL , we make the first attempt to theoretically understand DQN , which can be cast as an extension of the classical Q-learning algorithm ( Watkins & Dayan , 1992 ) that uses deep neural network to approximate the action-value function . Although the algorithmic and statistical properties of the classical Q-learning algorithm are well-studied , theoretical analysis of DQN is highly challenging due to its differences in the following two aspects . First , in online gradient-based temporal-difference reinforcement learning algorithms , approximating the action-value function often leads to instability . Baird ( 1995 ) proves that this is the case even with linear function approximation . The key technique to achieve stability in DQN is experience replay ( Lin , 1992 ; Mnih et al. , 2015 ) . In specific , a replay memory is used to store the trajectory of the Markov decision process ( MDP ) . At each iteration of DQN , a mini-batch of states , actions , rewards , and next states are sampled from the replay memory as observations to train the Q-network , which approximates the action-value function . The intuition behind experience replay is to achieve stability by breaking the temporal dependency among the observations used in the training of the deep neural network . Second , in addition to the aforementioned Q-network , DQN uses another neural network named target network to obtain an unbiased estimator of the mean-squared Bellman error used in training the Q-network . The target network is synchronized with the Q-network after each period of iterations , which leads to a coupling between the two networks . Moreover , even if we fix the target network and focus on updating the Q-network , the subproblem of training a neural network still remains less well-understood in theory . In this paper , we focus on a slight simplification of DQN , which is amenable to theoretical analysis while fully capturing the above two aspects . In specific , we simplify the technique of experience replay with an independence assumption , and focus on deep neural networks with rectified linear units ( ReLU ) ( Nair & Hinton , 2010 ) and large batch size . Under this setting , DQN is reduced to the neural fitted Q-iteration ( FQI ) algorithm ( Riedmiller , 2005 ) and the technique of target network can be cast as the value iteration . More importantly , by adapting the approximation results for ReLU networks to the analysis of Bellman operator , we establish the algorithmic and statistical rates of convergence for the iterative policy sequence obtained by DQN . As shown in the main results in §3 , the statistical error characterizes the bias and variance that arise from approximating the action-value function using neural network , while the algorithmic error geometrically decays to zero as the number of iteration goes to infinity . Our contribution is two-fold . First , we establish the algorithmic and statistical errors of the neural FQI algorithm , which can be viewed as a slight simplification of DQN . Under mild assumptions , our results show that the proposed algorithm obtains a sequence of Q-networks that geometrically converges to the optimal action-value function up to an intrinsic statistical error induced by the approximation bias of ReLU network and finite sample size . Second , as a byproduct , our analysis justifies the techniques of experience replay and target network used in DQN , where the latter can be viewed as the value iteration . In addition , we also extend our algorithm to zero-sum Markov games . Due to space limit , we defer these results to the appendix . Notation . For a measurable space with domain S , we denote by B ( S , V ) the set of measurable functions on S that are bounded by V in absolute value . Let P ( S ) be the set of all probability measures over S. For any ν ∈ P ( S ) and any measurable function f : S → R , we denote by ‖f‖p , ν the ` p-norm of f with respect to measure ν for p ≥ 1 . In addition , for simplicity , we write ‖f‖ν for ‖f‖2 , ν . In addition , let { f ( n ) , g ( n ) } n≥1 be two positive series . We write f ( n ) . g ( n ) if there exists a constant C such that f ( n ) ≤ C · g ( n ) for all n larger than some n0 ∈ N. In addition , we write f ( n ) g ( n ) if f ( n ) . g ( n ) and g ( n ) . f ( n ) . 2 BACKGROUND . In this section , we introduce the background . We first lay out the formulation of the reinforcement learning problem , and then define the family of ReLU neural networks . 2.1 REINFORCEMENT LEARNING . A discounted Markov decision process is defined by a tuple ( S , A , P , R , γ ) . Here S is the set of all states , which can be countable or uncountable , A is the set of all actions , P : S × A → P ( S ) is the Markov transition kernel , R : S × A → P ( R ) is the distribution of the immediate reward , and γ ∈ ( 0 , 1 ) is the discount factor . In specific , upon taking any action a ∈ A at any state s ∈ S , P ( · |s , a ) defines the probability distribution of the next state and R ( · |s , a ) is the distribution of the immediate reward . Moreover , for regularity , we further assume that S is a compact subset of Rd which can be infinite , A = { a1 , a2 , . . . , aM } has finite cardinality M , and the rewards are uniformly bounded by Rmax , i.e. , R ( · |s , a ) is supported on [ −Rmax , Rmax ] for any s ∈ S and a ∈ A . A policy π : S → P ( A ) for the MDP maps any state s ∈ S to a probability distribution π ( · | s ) over A . For policy π , the corresponding value function V π : S → R is defined as the cumulative discounted reward obtained by when the actions are executed according to π , that is , V π ( s ) = E [ ∞∑ t=0 γt ·Rt ∣∣∣∣S0 = s , At ∼ π ( · |St ) , St+1 ∼ P ( · |St , At ) ] . ( 2.1 ) Similarly , the action-value function Qπ : S ×A → R is defined as Qπ ( s , a ) = E [ ∞∑ t=0 γt ·Rt ∣∣∣∣S0 = s , A0 = a , At ∼ π ( · |St ) , St+1 ∼ P ( · |St , At ) ] . ( 2.2 ) For any given action-value function Q : S × A → R , define the one-step greedy policy πQ as any policy that selects the action with the largest Q-value , that is , for any s ∈ S , πQ ( · |s ) satisfies πQ ( a | s ) = 0 if Q ( s , a ) 6= max a′∈A Q ( s , a′ ) . ( 2.3 ) Moreover , we define operator Pπ by ( PπQ ) ( s , a ) = E [ Q ( S′ , A′ ) ∣∣S′ ∼ P ( · | s , a ) , A′ ∼ π ( · |S′ ) ] , ( 2.4 ) and define the Bellman operator Tπ by ( TπQ ) ( s , a ) = r ( s , a ) + γ · ( PπQ ) ( s , a ) , where r ( s , a ) =∫ rR ( dr | s , a ) is the expected reward obtained at state s when taking action a . Then it can be verified that Qπ is the unique fixed point of Tπ . The goal of reinforcement learning is to find the optimal policy , which achieves the largest cumulative reward . To characterize optimality , we define optimal action-value function Q∗ as Q∗ ( s , a ) = sup π Qπ ( s , a ) , ( 2.5 ) where the supremum is taken over all policies . Based on Q∗ , we define the optimal policy π∗ as any policy that is greedy with respect to Q∗ . It can be shown that Q∗ = Qπ ∗ . Finally , we define the Bellman optimality operator T via ( TQ ) ( s , a ) = r ( s , a ) + γ · E [ max a′∈A Q ( S′ , a′ ) ∣∣S′ ∼ P ( · | s , a ) ] . ( 2.6 ) Then we have Bellman optimality equation TQ∗ = Q∗ .

Summary: The authors provide a theoretical analysis of deep Q-learning based on the neural fitted Q-iteration (FQI) algorithm [1]. Their analysis justifies the techniques of experience replay and target network, both of which are critical to the empirical success of DQN. Moreover, the authors establish the algorithmic and statistical errors of the neural FQI algorithm.