Convergent and Efficient Deep Q Learning Algorithm

1 INTRODUCTION . With the development of deep learning , reinforcement learning ( RL ) that utilizes deep neural networks has demonstrated great success recently , finding applications in various fields including robotics , games , and scientific research ( Levine et al. , 2018 ; Berner et al. , 2019 ; Fösel et al. , 2018 ; Wang et al. , 2020 ) . One of the most efficient RL strategy is Q-learning ( Watkins , 1989 ) , and the combination of Q-learning and deep learning leads to the DQN algorithms ( Mnih et al. , 2015 ; Hessel et al. , 2018 ; Riedmiller , 2005 ) , which hold records on many difficult RL tasks ( Badia et al. , 2020 ) . However , unlike supervised learning , Q-learning , or more generally temporal difference ( TD ) learning , does not guarantee convergence when function approximations such as neural networks are used , and as a result , their success is actually empirical , and the performance relies heavily on hyperparameter tuning and technical details involved . This happens because the agent uses its own prediction to construct the learning objective , a.k.a . bootstrapping , and as it generalizes , its predictions over different data interfere with each other , which can make its learning objective unstable in the course of training and potentially lead to instability and divergence . This non-convergence problem was pointed out decades ago by the pioneering works of Baird ( 1995 ) and Tsitsiklis & Van Roy ( 1997 ) , and it has been empirically investigated for DQN by Van Hasselt et al . ( 2018 ) . We have also observed the divergence of DQN in our experiments , as in Fig . 6 . The non-convergence problem often shows up as instability in practice and it places significant obstacles to the application of DQN to complicated tasks . It makes the training with deeper neural networks more difficult , limits the time horizon for planning , and makes the results sometimes unstable and sensitive to hyperparameters . This state of affairs is not satisfactory especially for those scientific applications that require convergence and generality . Although convergent gradient-based methods have also been proposed ( Sutton et al. , 2009 ; Bhatnagar et al. , 2009 ; Feng et al. , 2019 ; Ghiassian et al. , 2020 ) , they can not easily be used with deep non-linear neural networks as they either require linearity or involve computationally heavy operations , and they often show worse empirical performance compared with TD methods . In this work , we show that the above-mentioned gradient-based methods actually have inherent problems in learning dynamics which hamper efficient learning , and we propose a convergent DQN ( C-DQN ) algorithm by modifying the loss of DQN . Because an increase of loss upon updating the target network of DQN is a necessary condition for its divergence , we construct a loss that does not increase upon the update of the target network , and therefore , the proposed algorithm converges in the sense that the loss monotonically decreases . In Sec . 2 we present the background . In Sec . 3 we discuss the inefficiency issues in the previous gradient-based methods and demonstrate using toy problems . In Sec . 4 we propose C-DQN and show its convergence . In Sec . 5 we show the results of C-DQN on the Atari 2600 benchmark ( Bellemare et al. , 2013 ) and in Sec . 6 we present the conclusion and future prospect . To our knowledge , the proposed C-DQN algorithm is the first convergent RL method that is sufficiently efficient and scalable to obtain successful results on the standard Atari 2600 benchmark using deep neural networks , showing its efficacy in dealing with realistic and complicated problems . 2 BACKGROUND . Reinforcement learning involves a Markov decision process ( MDP ) , where the state st of an environment at time step tmakes a transition to the next state st+1 conditioned on the action of the agent at at time t , producing a reward rt depending on the states . The process can terminate at terminal states sT , and the transition of states can be either probabilistic or deterministic . The goal is to find a policy π ( s ) to determine the actions at+i ∼ π ( st+i ) in order to maximizes the return ∑T−t i=0 rt+i , i.e. , the sum of future rewards . In practice , a discounted return ∑T−t i=0 γ irt+i is often used instead , with the discount factor γ < 1 and γ ≈ 1 , so that the expression is convergent for T →∞ and that rewards far into the future can be ignored , giving an effective time horizon 11−γ . The value function is defined as the expected return for a state st following a policy π , and the Q function is defined as the expected return for a state-action pair ( st , at ) : Vπ ( st ) = Eat , { ( st+i , at+i ) } T−ti=1 [ T−t∑ i=0 γirt+i ] , Qπ ( st , at ) = E { ( st+i , at+i ) } T−ti=1 [ T−t∑ i=0 γirt+i ] , ( 1 ) with at+i ∼ π ( st+i ) in the evaluation of the expectation . When the Q function is maximized by a policy , we say that the policy is optimal and denote the Q function and the policy by Q∗ and π∗ , respectively . The optimality implies that Q∗ satisfies the Bellman equation ( Sutton & Barto , 2018 ) Q∗ ( st , at ) = rt + γEst+1 [ max a′ Q∗ ( st+1 , a ′ ) ] . ( 2 ) The policy π∗ is greedy with respect to Q∗ , i.e . π∗ ( s ) = arg maxa′ Q ∗ ( s , a′ ) . Q-learning uses this recursive relation to learn Q∗ . In this work we only consider the deterministic case and drop the notation Est+1 [ · ] where appropriate . When the space of state-action pairs is small and finite , we can write down the values of an arbitrarily initialized Q function for all state-action pairs into a table , and iterate over the values using ∆Q ( st , at ) = α ( rt + γmax a′ Q ( st+1 , a ′ ) −Q ( st , at ) ) , ( 3 ) where α is the learning rate . This is called Q-table learning and it guarantees convergence to Q∗ . If the space of ( s , a ) is large and Q-table learning is impossible , a function approximation is used instead , representing the Q function as Qθ with learnable parameter θ . The learning rule is ∆θ = α∇θQθ ( st , at ) ( rt + γmax a′ Qθ ( st+1 , a ′ ) −Qθ ( st , at ) ) , ( 4 ) which can be interpreted as modifying the value of Qθ ( st , at ) following the gradient so that Qθ ( st , at ) approaches the target value rt + γmaxa′ Qθ ( st+1 , a′ ) . However , this iteration may not converge , because the term maxa′ Qθ ( st+1 , a′ ) is also θ-dependent and may change together with Qθ ( st , at ) . Specifically , an exponential divergence occurs if γ∇θQθ ( st , at ) · ∇θ maxa′ Qθ ( st+1 , a′ ) > ||∇θQθ ( st , at ) ||2 is always satisfied and the value of maxa′ Qθ ( st+1 , a′ ) is not constrained by other means.1 This can be a serious issue for realistic tasks , because the adjacent states st and st+1 often have similar representations and∇θQθ ( st , · ) is close to∇θQθ ( st+1 , · ) . The DQN algorithm uses a deep neural network with parameters θ as Qθ ( Mnih et al. , 2015 ) , and to stabilize learning , it introduces a target network with parameters θ̃ , and replace the term maxa′ Qθ ( st+1 , a ′ ) by maxa′ Qθ̃ ( st+1 , a ′ ) , so that the target value rt + γmaxa′ Qθ̃ ( st+1 , a ′ ) does 1This can be shown by checking the Bellman error δt : = rt + γmaxa′ Qθ ( st+1 , a′ ) − Qθ ( st , at ) , for which we have ∆δt = αδt ( γ∇θQθ ( st , at ) ·maxa′ Qθ ( st+1 , a′ ) − ||∇θQθ ( st , at ) ||2 ) up to the first order of ∆θ following Eq . ( 4 ) . As ∆δt is proportional to δt with the same sign , it can increase exponentially . not change simultaneously with Qθ . The target network θ̃ is then updated by copying from θ for every few thousand iterations of θ . This technique reduces fluctuations in the target value and dramatically improves the stability of learning , and with the use of offline sampling and adaptive optimizers , it can learn various tasks such as video games and simulated robotic control ( Mnih et al. , 2015 ; Lillicrap et al. , 2015 ) . Nevertheless , the introduction of the target network θ̃ is not well-principled , and it does not really preclude the possibility of divergence . As a result , DQN sometimes requires a significant amount of hyperparameter tuning in order to work well for a new task , and in some cases , the instability in learning can be hard to diagnose or remove , and usually one can not use a discount factor γ that is very close to 1 . In an attempt to solve this problem , Durugkar & Stone ( 2017 ) considered only updating θ in a direction that is perpendicular to ∇θ maxa′ Qθ ( st+1 , a′ ) ; however , this strategy is not satisfactory in general and can lead to poor performance , as shown in Pohlen et al . ( 2018 ) . One way of approaching this problem is to consider the mean squared Bellman error ( MSBE ) , which is originally proposed by Baird ( 1995 ) and called the residual gradient ( RG ) algorithm . The Bellman error , or Bellman residual , TD error , is given by δt ( θ ) : = rt + γmaxa′ Qθ ( st+1 , a′ ) − Qθ ( st , at ) . Given a dataset S of state-action data , δt is a function of θ , and we can minimize the MSBE loss LMSBE ( θ ) : = E [ |δ ( θ ) |2 ] = 1 |S| ∑ ( st , at , rt , st+1 ) ∈S ∣∣∣Qθ ( st , at ) − rt − γmax a′ Qθ ( st+1 , a ′ ) ∣∣∣2 , ( 5 ) and in practice the loss is minimized via gradient descent . If LMSBE becomes zero , we have δt ≡ 0 and the Bellman equation is satisfied , implying Qθ = Q∗ . Given a fixed dataset S , the convergence of the learning process simply follows the convergence of the optimization of the loss . This strategy can be used with neural networks straightforwardly . There have also been many improvements on this strategy including Sutton et al . ( 2008 ; 2009 ) ; Bhatnagar et al . ( 2009 ) ; Dai et al . ( 2018 ) ; Feng et al . ( 2019 ) ; Ghiassian et al . ( 2020 ) ; Touati et al . ( 2018 ) , and they are often referred to as gradientTD methods . Many of them have focused on how to evaluate the expectation term in Eq . ( 2 ) and make it converge to the same solution found by TD , or Q-learning . However , most of these methods often do not work well for difficult problems , and few of them have been successfully demonstrated on standard RL benchmarks , especially the Atari 2600 benchmark . In the following we refer to the strategy of simply minimizing LMSBE as the RG algorithm , or RG learning .

Summary: The paper argues that the DQN and its algorithm and its variants do not guarantee convergence and that they can diverge in realistic settings.   The authors develop a new technique that guarantees convergence of DQN. The authors show that in the case of very large discount factors (such as 0.9998) this algorithm outperforms baseline algorithms on certain Atari games.