Convergent and Efficient Deep Q Learning Algorithm

1 INTRODUCTION . With the development of deep learning , reinforcement learning ( RL ) that utilizes deep neural networks has demonstrated great success recently , finding applications in various fields including robotics , games , and scientific research ( Levine et al. , 2018 ; Berner et al. , 2019 ; Fösel et al. , 2018 ; Wang et al. , 2020 ) . One of the most efficient RL strategy is Q-learning ( Watkins , 1989 ) , and the combination of Q-learning and deep learning leads to the DQN algorithms ( Mnih et al. , 2015 ; Hessel et al. , 2018 ; Riedmiller , 2005 ) , which hold records on many difficult RL tasks ( Badia et al. , 2020 ) . However , unlike supervised learning , Q-learning , or more generally temporal difference ( TD ) learning , does not guarantee convergence when function approximations such as neural networks are used , and as a result , their success is actually empirical , and the performance relies heavily on hyperparameter tuning and technical details involved . This happens because the agent uses its own prediction to construct the learning objective , a.k.a . bootstrapping , and as it generalizes , its predictions over different data interfere with each other , which can make its learning objective unstable in the course of training and potentially lead to instability and divergence . This non-convergence problem was pointed out decades ago by the pioneering works of Baird ( 1995 ) and Tsitsiklis & Van Roy ( 1997 ) , and it has been empirically investigated for DQN by Van Hasselt et al . ( 2018 ) . We have also observed the divergence of DQN in our experiments , as in Fig . 6 . The non-convergence problem often shows up as instability in practice and it places significant obstacles to the application of DQN to complicated tasks . It makes the training with deeper neural networks more difficult , limits the time horizon for planning , and makes the results sometimes unstable and sensitive to hyperparameters . This state of affairs is not satisfactory especially for those scientific applications that require convergence and generality . Although convergent gradient-based methods have also been proposed ( Sutton et al. , 2009 ; Bhatnagar et al. , 2009 ; Feng et al. , 2019 ; Ghiassian et al. , 2020 ) , they can not easily be used with deep non-linear neural networks as they either require linearity or involve computationally heavy operations , and they often show worse empirical performance compared with TD methods . In this work , we show that the above-mentioned gradient-based methods actually have inherent problems in learning dynamics which hamper efficient learning , and we propose a convergent DQN ( C-DQN ) algorithm by modifying the loss of DQN . Because an increase of loss upon updating the target network of DQN is a necessary condition for its divergence , we construct a loss that does not increase upon the update of the target network , and therefore , the proposed algorithm converges in the sense that the loss monotonically decreases . In Sec . 2 we present the background . In Sec . 3 we discuss the inefficiency issues in the previous gradient-based methods and demonstrate using toy problems . In Sec . 4 we propose C-DQN and show its convergence . In Sec . 5 we show the results of C-DQN on the Atari 2600 benchmark ( Bellemare et al. , 2013 ) and in Sec . 6 we present the conclusion and future prospect . To our knowledge , the proposed C-DQN algorithm is the first convergent RL method that is sufficiently efficient and scalable to obtain successful results on the standard Atari 2600 benchmark using deep neural networks , showing its efficacy in dealing with realistic and complicated problems . 2 BACKGROUND . Reinforcement learning involves a Markov decision process ( MDP ) , where the state st of an environment at time step tmakes a transition to the next state st+1 conditioned on the action of the agent at at time t , producing a reward rt depending on the states . The process can terminate at terminal states sT , and the transition of states can be either probabilistic or deterministic . The goal is to find a policy π ( s ) to determine the actions at+i ∼ π ( st+i ) in order to maximizes the return ∑T−t i=0 rt+i , i.e. , the sum of future rewards . In practice , a discounted return ∑T−t i=0 γ irt+i is often used instead , with the discount factor γ < 1 and γ ≈ 1 , so that the expression is convergent for T →∞ and that rewards far into the future can be ignored , giving an effective time horizon 11−γ . The value function is defined as the expected return for a state st following a policy π , and the Q function is defined as the expected return for a state-action pair ( st , at ) : Vπ ( st ) = Eat , { ( st+i , at+i ) } T−ti=1 [ T−t∑ i=0 γirt+i ] , Qπ ( st , at ) = E { ( st+i , at+i ) } T−ti=1 [ T−t∑ i=0 γirt+i ] , ( 1 ) with at+i ∼ π ( st+i ) in the evaluation of the expectation . When the Q function is maximized by a policy , we say that the policy is optimal and denote the Q function and the policy by Q∗ and π∗ , respectively . The optimality implies that Q∗ satisfies the Bellman equation ( Sutton & Barto , 2018 ) Q∗ ( st , at ) = rt + γEst+1 [ max a′ Q∗ ( st+1 , a ′ ) ] . ( 2 ) The policy π∗ is greedy with respect to Q∗ , i.e . π∗ ( s ) = arg maxa′ Q ∗ ( s , a′ ) . Q-learning uses this recursive relation to learn Q∗ . In this work we only consider the deterministic case and drop the notation Est+1 [ · ] where appropriate . When the space of state-action pairs is small and finite , we can write down the values of an arbitrarily initialized Q function for all state-action pairs into a table , and iterate over the values using ∆Q ( st , at ) = α ( rt + γmax a′ Q ( st+1 , a ′ ) −Q ( st , at ) ) , ( 3 ) where α is the learning rate . This is called Q-table learning and it guarantees convergence to Q∗ . If the space of ( s , a ) is large and Q-table learning is impossible , a function approximation is used instead , representing the Q function as Qθ with learnable parameter θ . The learning rule is ∆θ = α∇θQθ ( st , at ) ( rt + γmax a′ Qθ ( st+1 , a ′ ) −Qθ ( st , at ) ) , ( 4 ) which can be interpreted as modifying the value of Qθ ( st , at ) following the gradient so that Qθ ( st , at ) approaches the target value rt + γmaxa′ Qθ ( st+1 , a′ ) . However , this iteration may not converge , because the term maxa′ Qθ ( st+1 , a′ ) is also θ-dependent and may change together with Qθ ( st , at ) . Specifically , an exponential divergence occurs if γ∇θQθ ( st , at ) · ∇θ maxa′ Qθ ( st+1 , a′ ) > ||∇θQθ ( st , at ) ||2 is always satisfied and the value of maxa′ Qθ ( st+1 , a′ ) is not constrained by other means.1 This can be a serious issue for realistic tasks , because the adjacent states st and st+1 often have similar representations and∇θQθ ( st , · ) is close to∇θQθ ( st+1 , · ) . The DQN algorithm uses a deep neural network with parameters θ as Qθ ( Mnih et al. , 2015 ) , and to stabilize learning , it introduces a target network with parameters θ̃ , and replace the term maxa′ Qθ ( st+1 , a ′ ) by maxa′ Qθ̃ ( st+1 , a ′ ) , so that the target value rt + γmaxa′ Qθ̃ ( st+1 , a ′ ) does 1This can be shown by checking the Bellman error δt : = rt + γmaxa′ Qθ ( st+1 , a′ ) − Qθ ( st , at ) , for which we have ∆δt = αδt ( γ∇θQθ ( st , at ) ·maxa′ Qθ ( st+1 , a′ ) − ||∇θQθ ( st , at ) ||2 ) up to the first order of ∆θ following Eq . ( 4 ) . As ∆δt is proportional to δt with the same sign , it can increase exponentially . not change simultaneously with Qθ . The target network θ̃ is then updated by copying from θ for every few thousand iterations of θ . This technique reduces fluctuations in the target value and dramatically improves the stability of learning , and with the use of offline sampling and adaptive optimizers , it can learn various tasks such as video games and simulated robotic control ( Mnih et al. , 2015 ; Lillicrap et al. , 2015 ) . Nevertheless , the introduction of the target network θ̃ is not well-principled , and it does not really preclude the possibility of divergence . As a result , DQN sometimes requires a significant amount of hyperparameter tuning in order to work well for a new task , and in some cases , the instability in learning can be hard to diagnose or remove , and usually one can not use a discount factor γ that is very close to 1 . In an attempt to solve this problem , Durugkar & Stone ( 2017 ) considered only updating θ in a direction that is perpendicular to ∇θ maxa′ Qθ ( st+1 , a′ ) ; however , this strategy is not satisfactory in general and can lead to poor performance , as shown in Pohlen et al . ( 2018 ) . One way of approaching this problem is to consider the mean squared Bellman error ( MSBE ) , which is originally proposed by Baird ( 1995 ) and called the residual gradient ( RG ) algorithm . The Bellman error , or Bellman residual , TD error , is given by δt ( θ ) : = rt + γmaxa′ Qθ ( st+1 , a′ ) − Qθ ( st , at ) . Given a dataset S of state-action data , δt is a function of θ , and we can minimize the MSBE loss LMSBE ( θ ) : = E [ |δ ( θ ) |2 ] = 1 |S| ∑ ( st , at , rt , st+1 ) ∈S ∣∣∣Qθ ( st , at ) − rt − γmax a′ Qθ ( st+1 , a ′ ) ∣∣∣2 , ( 5 ) and in practice the loss is minimized via gradient descent . If LMSBE becomes zero , we have δt ≡ 0 and the Bellman equation is satisfied , implying Qθ = Q∗ . Given a fixed dataset S , the convergence of the learning process simply follows the convergence of the optimization of the loss . This strategy can be used with neural networks straightforwardly . There have also been many improvements on this strategy including Sutton et al . ( 2008 ; 2009 ) ; Bhatnagar et al . ( 2009 ) ; Dai et al . ( 2018 ) ; Feng et al . ( 2019 ) ; Ghiassian et al . ( 2020 ) ; Touati et al . ( 2018 ) , and they are often referred to as gradientTD methods . Many of them have focused on how to evaluate the expectation term in Eq . ( 2 ) and make it converge to the same solution found by TD , or Q-learning . However , most of these methods often do not work well for difficult problems , and few of them have been successfully demonstrated on standard RL benchmarks , especially the Atari 2600 benchmark . In the following we refer to the strategy of simply minimizing LMSBE as the RG algorithm , or RG learning .

Summary: This paper studies several issues of DQN (basically FQI, a version of fixed-point iteration) and empirical Bellman error minimization (has a clear loss function). Several issues of MSBE are pointed out. The authors propose a new approach by replacing the DQN loss by the maximum of DQN loss and MSBE loss. Several experiments on Atari games support the argument.