A Theoretical Analysis of  Deep Q-Learning

Despite the great empirical success of deep reinforcement learning , its theoretical foundation is less well understood . In this work , we make the first attempt to theoretically understand the deep Q-network ( DQN ) algorithm ( Mnih et al. , 2015 ) from both algorithmic and statistical perspectives . In specific , we focus on the fitted Q iteration ( FQI ) algorithm with deep neural networks , which is a slight simplification of DQN that captures the tricks of experience replay and target network used in DQN . Under mild assumptions , we establish the algorithmic and statistical rates of convergence for the action-value functions of the iterative policy sequence obtained by FQI . In particular , the statistical error characterizes the bias and variance that arise from approximating the action-value function using deep neural network , while the algorithmic error converges to zero at a geometric rate . As a byproduct , our analysis provides justifications for the techniques of experience replay and target network , which are crucial to the empirical success of DQN . Furthermore , as a simple extension of DQN , we propose the Minimax-DQN algorithm for zero-sum Markov game with two players , which is deferred to the appendix due to space limitations . 1 INTRODUCTION . Reinforcement learning ( RL ) attacks the multi-stage decision-making problems by interacting with the environment and learning from the experiences . With the breakthrough in deep learning , deep reinforcement learning ( DRL ) demonstrates tremendous success in solving highly challenging problems , such as the game of Go ( Silver et al. , 2016 ; 2017 ) , robotics ( Kober & Peters , 2012 ) , and dialogue systems ( Chen et al. , 2017 ) . In DRL , the value or policy functions are often represented as deep neural networks and the related deep learning techniques can be readily applied . For example , deep Q-network ( DQN ) ( Mnih et al. , 2015 ) , asynchronous advantage actor-critic ( A3C ) and ( Mnih et al. , 2016 ) demonstrate superhuman performance in various applications and become standard algorithms for artificial intelligence . Despite its great empirical success , there exists a gap between the theory and practice of DRL . In particular , most existing theoretical work on reinforcement learning focuses on the tabular case where the state and action spaces are finite , or the case where the value function is linear . Under these restrictive settings , the algorithmic and statistical perspectives of reinforcement learning are well-understood via the tools developed for convex optimization and linear regression . However , in presence of nonlinear function approximators such as deep neural network , the theoretical analysis of reinforcement learning becomes intractable as it involves solving a highly nonconvex statistical optimization problem . To bridge such a gap in DRL , we make the first attempt to theoretically understand DQN , which can be cast as an extension of the classical Q-learning algorithm ( Watkins & Dayan , 1992 ) that uses deep neural network to approximate the action-value function . Although the algorithmic and statistical properties of the classical Q-learning algorithm are well-studied , theoretical analysis of DQN is highly challenging due to its differences in the following two aspects . First , in online gradient-based temporal-difference reinforcement learning algorithms , approximating the action-value function often leads to instability . Baird ( 1995 ) proves that this is the case even with linear function approximation . The key technique to achieve stability in DQN is experience replay ( Lin , 1992 ; Mnih et al. , 2015 ) . In specific , a replay memory is used to store the trajectory of the Markov decision process ( MDP ) . At each iteration of DQN , a mini-batch of states , actions , rewards , and next states are sampled from the replay memory as observations to train the Q-network , which approximates the action-value function . The intuition behind experience replay is to achieve stability by breaking the temporal dependency among the observations used in the training of the deep neural network . Second , in addition to the aforementioned Q-network , DQN uses another neural network named target network to obtain an unbiased estimator of the mean-squared Bellman error used in training the Q-network . The target network is synchronized with the Q-network after each period of iterations , which leads to a coupling between the two networks . Moreover , even if we fix the target network and focus on updating the Q-network , the subproblem of training a neural network still remains less well-understood in theory . In this paper , we focus on a slight simplification of DQN , which is amenable to theoretical analysis while fully capturing the above two aspects . In specific , we simplify the technique of experience replay with an independence assumption , and focus on deep neural networks with rectified linear units ( ReLU ) ( Nair & Hinton , 2010 ) and large batch size . Under this setting , DQN is reduced to the neural fitted Q-iteration ( FQI ) algorithm ( Riedmiller , 2005 ) and the technique of target network can be cast as the value iteration . More importantly , by adapting the approximation results for ReLU networks to the analysis of Bellman operator , we establish the algorithmic and statistical rates of convergence for the iterative policy sequence obtained by DQN . As shown in the main results in §3 , the statistical error characterizes the bias and variance that arise from approximating the action-value function using neural network , while the algorithmic error geometrically decays to zero as the number of iteration goes to infinity . Our contribution is two-fold . First , we establish the algorithmic and statistical errors of the neural FQI algorithm , which can be viewed as a slight simplification of DQN . Under mild assumptions , our results show that the proposed algorithm obtains a sequence of Q-networks that geometrically converges to the optimal action-value function up to an intrinsic statistical error induced by the approximation bias of ReLU network and finite sample size . Second , as a byproduct , our analysis justifies the techniques of experience replay and target network used in DQN , where the latter can be viewed as the value iteration . In addition , we also extend our algorithm to zero-sum Markov games . Due to space limit , we defer these results to the appendix . Notation . For a measurable space with domain S , we denote by B ( S , V ) the set of measurable functions on S that are bounded by V in absolute value . Let P ( S ) be the set of all probability measures over S. For any ν ∈ P ( S ) and any measurable function f : S → R , we denote by ‖f‖p , ν the ` p-norm of f with respect to measure ν for p ≥ 1 . In addition , for simplicity , we write ‖f‖ν for ‖f‖2 , ν . In addition , let { f ( n ) , g ( n ) } n≥1 be two positive series . We write f ( n ) . g ( n ) if there exists a constant C such that f ( n ) ≤ C · g ( n ) for all n larger than some n0 ∈ N. In addition , we write f ( n ) g ( n ) if f ( n ) . g ( n ) and g ( n ) . f ( n ) . 2 BACKGROUND . In this section , we introduce the background . We first lay out the formulation of the reinforcement learning problem , and then define the family of ReLU neural networks . 2.1 REINFORCEMENT LEARNING . A discounted Markov decision process is defined by a tuple ( S , A , P , R , γ ) . Here S is the set of all states , which can be countable or uncountable , A is the set of all actions , P : S × A → P ( S ) is the Markov transition kernel , R : S × A → P ( R ) is the distribution of the immediate reward , and γ ∈ ( 0 , 1 ) is the discount factor . In specific , upon taking any action a ∈ A at any state s ∈ S , P ( · |s , a ) defines the probability distribution of the next state and R ( · |s , a ) is the distribution of the immediate reward . Moreover , for regularity , we further assume that S is a compact subset of Rd which can be infinite , A = { a1 , a2 , . . . , aM } has finite cardinality M , and the rewards are uniformly bounded by Rmax , i.e. , R ( · |s , a ) is supported on [ −Rmax , Rmax ] for any s ∈ S and a ∈ A . A policy π : S → P ( A ) for the MDP maps any state s ∈ S to a probability distribution π ( · | s ) over A . For policy π , the corresponding value function V π : S → R is defined as the cumulative discounted reward obtained by when the actions are executed according to π , that is , V π ( s ) = E [ ∞∑ t=0 γt ·Rt ∣∣∣∣S0 = s , At ∼ π ( · |St ) , St+1 ∼ P ( · |St , At ) ] . ( 2.1 ) Similarly , the action-value function Qπ : S ×A → R is defined as Qπ ( s , a ) = E [ ∞∑ t=0 γt ·Rt ∣∣∣∣S0 = s , A0 = a , At ∼ π ( · |St ) , St+1 ∼ P ( · |St , At ) ] . ( 2.2 ) For any given action-value function Q : S × A → R , define the one-step greedy policy πQ as any policy that selects the action with the largest Q-value , that is , for any s ∈ S , πQ ( · |s ) satisfies πQ ( a | s ) = 0 if Q ( s , a ) 6= max a′∈A Q ( s , a′ ) . ( 2.3 ) Moreover , we define operator Pπ by ( PπQ ) ( s , a ) = E [ Q ( S′ , A′ ) ∣∣S′ ∼ P ( · | s , a ) , A′ ∼ π ( · |S′ ) ] , ( 2.4 ) and define the Bellman operator Tπ by ( TπQ ) ( s , a ) = r ( s , a ) + γ · ( PπQ ) ( s , a ) , where r ( s , a ) =∫ rR ( dr | s , a ) is the expected reward obtained at state s when taking action a . Then it can be verified that Qπ is the unique fixed point of Tπ . The goal of reinforcement learning is to find the optimal policy , which achieves the largest cumulative reward . To characterize optimality , we define optimal action-value function Q∗ as Q∗ ( s , a ) = sup π Qπ ( s , a ) , ( 2.5 ) where the supremum is taken over all policies . Based on Q∗ , we define the optimal policy π∗ as any policy that is greedy with respect to Q∗ . It can be shown that Q∗ = Qπ ∗ . Finally , we define the Bellman optimality operator T via ( TQ ) ( s , a ) = r ( s , a ) + γ · E [ max a′∈A Q ( S′ , a′ ) ∣∣S′ ∼ P ( · | s , a ) ] . ( 2.6 ) Then we have Bellman optimality equation TQ∗ = Q∗ .

Summary: This paper provides an analysis of fitted Q-iteration in the off-policy reinforcement learning setting, for the setup where the value function class of interest is a class of neural networks. The provide bounds on the rate at which fitted Q iteration converges to a near-optimal policy under the assumption that the transition dynamics satisfy a certain notion of Holder smoothness. This result is motivated by the problem of understanding why deep Q-learning works, which the authors relate to the problem above via certain simplifying assumptions. The authors also extend this result to give similar guarantees for two-player zero-sum stochastic games.