Decorrelated Double Q-learning

1 INTRODUCTION . Q-learning Watkins & Dayan ( 1992 ) as a model free reinforcement learning approach has gained popularity , especially under the advance of deep neural networks Mnih et al . ( 2013 ) . In general , it combines the neural network approximators with the actor-critic architectures Witten ( 1977 ) ; Konda & Tsitsiklis ( 1999 ) , which has an actor network to control how the agent behaves and a critic to evaluate how good the action taken is . The Deep Q-Network ( DQN ) algorithm Mnih et al . ( 2013 ) firstly applied the deep neural network to approximate the action-value function in Q-learning and shown remarkably good and stable results by introducing a target network and Experience Replay buffer to stabilize the training . Lillicrap et al . proposes DDPG Lillicrap et al . ( 2015 ) , which extends Q-learning to handle continuous action space with target networks . Except the training stability , another issue Q-learning suffered is overestimation bias , which was first investigated in Thrun & Schwartz ( 1993 ) . Because of the noise in function approximation , the maximum operator in Q-learning can lead to overestimation of state-action values . And , the overestimation property is also observed in deterministic continuous policy control Silver & Lever ( 2014 ) . In particular , with the imprecise function approximation , the maximization of a noisy value will induce overestimation to the action value function . This inaccuracy could be even worse ( e.g . error accumulation ) under temporal difference learning Sutton & Barto ( 1998 ) , in which bootstrapping method is used to update the value function using the estimate of a subsequent state . Given overestimation bias caused by maximum operator of noise estimate , many methods have been proposed to address this issue . Double Q-learning van Hasselt ( 2010 ) mitigates the overestimation effect by introducing two independently critics to estimate the maximum value of a set of stochastic values . Averaged-DQN Anschel et al . ( 2017 ) takes the average of previously learned Q-values estimates , which results in a more stable training procedure , as well as reduces approximation error variance in the target values . Recently , Twin Delayed Deep Deterministic Policy Gradients ( TD3 ) Fujimoto et al . ( 2018 ) extends the Double Q-learning , by using the minimum of two critics to limit the overestimated bias in actor-critic network . A soft Q-learning algorithm Haarnoja et al . ( 2018 ) , called soft actor-critic , leverages the similar strategy as TD3 , while including the maximum entropy to balance exploration and exploitation . Maxmin Q-learning Lan et al . ( 2020 ) proposes the use of an ensembling scheme to handle overestimation bias in Q-Learning . This work suggests an alternative solution to the overestimation phenomena , called decorrelated double Q-learning , based on reducing the noise estimate in Q-values . On the one hand , we want to make the two value function approximators as independent as possible to mitigate overestima- tion bias . On the other hand , we should reduce the variance caused by imprecise estimate . Our decorrelated double Q-learning proposes an objective function to minimize the correlation of two critics , and meanwhile reduces the target approximation error variance with control variate methods . Finally , we provide experimental results on MuJoCo games and show significant improvement compared to competitive baselines . The paper is organized as follows . In Section 2 , we introduce reinforcement learning problems , notations and two existed Q-learning variants to address overestimation bias . Then we present our D2Q algorithm in Section 3 and also prove that in the limit , this algorithm converges to the optimal solution . In Section 4 we show the experimental results on MuJoCo continuous control tasks , and compare it to the current state of the art . Some related work and discussion is presented in Section 5 and finally Section 6 concludes the paper . 2 BACKGROUND . In this section , we introduce the reinforcement learning problems and Q-learning , as well as notions that will be used in the following sections . 2.1 PROBLEM SETTING AND NOTATIONS . We consider the model-free reinforcement learning problem ( i.e . optimal policy existed ) with sequential interactions between an agent and its environment Sutton & Barto ( 1998 ) in order to maximize a cumulative return . At every time step t , the agent selects an action at in the state st according its policy and receives a scalar reward rt ( st , at ) , and then transit to the next state st+1 . The problem is modeled as Markov decision process ( MDP ) with tuple : ( S , A , p ( s0 ) , p ( st+1|st , at ) , r ( st , at ) , γ ) . Here , S and A indicate the state and action space respectively , p ( s0 ) is the initial state distribution . p ( st+1|st , at ) is the state transition probability to st+1 given the current state st and action at , r ( st , at ) is reward from the environment after the agent taking action at in state st and γ is discount factor , which is necessary to decay the future rewards ensuring finite returns . We model the agent ’ s behavior with πθ ( a|s ) , which is a parametric distribution from a neural network . Suppose we have the finite length trajectory while the agent interacting with the environment . The return under the policy π for a trajectory τ = ( st , at ) T t=0 J ( θ ) = Eτ∼πθ ( τ ) [ r ( τ ) ] = Eτ∼πθ ( τ ) [ R T 0 ] = Eτ∼πθ ( τ ) [ T∑ t=0 γtr ( st , at ) ] ( 1 ) where πθ ( τ ) denotes the distribution of trajectories , p ( τ ) = π ( s0 , a0 , s1 , ... , sT , aT ) = p ( s0 ) T∏ t=0 πθ ( at|st ) p ( st+1|st , at ) ( 2 ) The goal of reinforcement learning is to learn a policy π which can maximize the expected returns θ = arg max θ J ( θ ) = arg maxEτ∼πθ ( τ ) [ R T 0 ] ( 3 ) The action-value function describes what the expected return of the agent is in state s and action a under the policy π . The advantage of action value function is to make actions explicit , so we can select actions even in the model-free environment . After taking an action at in state st and thereafter following policy π , the action value function is formatted as : Qπ ( st , at ) = Esi∼pπ , ai∼π [ Rt|st , at ] = Esi∼pπ , ai∼π [ T∑ i=t γ ( i−t ) r ( si , ai ) |st , at ] ( 4 ) To get the optimal value function , we can use the maximum over actions , denoted as Q∗ ( st , at ) = maxπ Q π ( st , at ) , and the corresponding optimal policy π can be easily derived by π∗ ( s ) ∈ arg maxat Q ∗ ( st , at ) . 2.2 Q-LEARNING . Q-learning , as an off-policy RL algorithm , has been extensively studied since it was proposed Watkins & Dayan ( 1992 ) . Suppose we use neural network parametrized by θQ to approximate Q-value in the continuous environment . To update Q-value function , we minimize the follow loss : L ( θQ ) = Esi∼pπ , ai∼π [ ( Q ( st , at ; θQ ) − yt ) 2 ] ( 5 ) where yt = r ( st , at ) + γmaxat+1 Q ( st+1 , at+1 ; θ Q ) is from Bellman equation , and its action at+1 is taken from frozen policy network ( actor ) to stabilizing the learning . In actor-critic methods , the policy π : S 7→ A , known as the actor with parameters θπ , can be updated through the chain rule in the deterministic policy gradient algorithm Silver & Lever ( 2014 ) ∇J ( θπ ) = Es∼pπ [ ∇aQ ( s , a ; θQ ) |a=π ( s ; θπ ) ∇θπ ( π ( s ; θπ ) ) ] ( 6 ) where Q ( s , a ) is the expected return while taking action a in state s , and following π after . One issue has attracted great attention is overestimation bias , which may exacerbate the situation into a more significant bias over the following updates if left unchecked . Moreover , an inaccurate value estimate may lead to poor policy updates . To address it , Double Q-learning van Hasselt ( 2010 ) use two independent critics q1 ( st , at ) and q2 ( st , at ) , where policy selection uses a different critic network than value estimation q1 ( st , at ) = r ( st , at ) + γq2 ( st+1 , arg max at+1 q1 ( st+1 , at+1 ; θ q1 ) ; θq2 ) q2 ( st , at ) = r ( st , at ) + γq1 ( st+1 , arg max at+1 q2 ( st+1 , at+1 ; θ q2 ) ; θq1 ) Recently , TD3 Fujimoto et al . ( 2018 ) uses the similar two q-value functions , but taking the minimum of them below : yt = r ( st , at ) + γmin ( q1 ( st+1 , π ( st+1 ) ) , q2 ( st+1 , π ( st+1 ) ) ) ( 7 ) Then the same square loss in Eq . 5 can be used to learn model parameters . 3 DECORRELATED DOUBLE Q-LEARNING . In this section , we present Decorrelated Double Q-learning ( D2Q ) for continuous action control with attempt to address overestimation bias . Similar to Double Q-learning , we use two q-value functions to approximate Q ( st , at ) . Our main contribution is to borrow the idea from control variates to decorrelate these two value functions , which can further reduce the overestimation risk . 3.1 Q-VALUE FUNCTION . Suppose we have two approximators q1 ( st , at ) and q2 ( st , at ) , D2Q uses the weighted difference of double q-value functions to approximate the action-value function at ( st , at ) . Thus , we define Q-value as following : Q ( st , at ) = q1 ( st , at ) − β ( q2 ( st , at ) − E ( q2 ( st , at ) ) ) ( 8 ) where q2 ( st , at ) −E ( q2 ( st , at ) ) is to model the noise in state st and action at , and β is the correlation coefficient of q1 ( st , at ) and q2 ( st , at ) . To understand the expectation E ( q2 ( st , at ) ) , it is the average over all possible runs . Thus , the weighted difference between q1 ( st , at ) and q2 ( st , at ) attempts to reduce the variance and remove the noise effects in Q-learning . To update q1 and q2 , we minimize the following loss : L ( θQ ) = Esi∼pπ , ai∼π [ ( q1 ( st , at ; θq1 ) − yt ) 2 ] + Esi∼pπ , ai∼π [ ( q2 ( st , at ; θq2 ) − yt ) 2 ] + λEsi∼pπ , ai∼π [ corr ( q1 ( st , at ; θq1 ) , q2 ( st , at ; θq2 ) ) ] 2 ( 9 ) where θQ = { θq1 , θq2 } , and yt can be defined as yt = r ( st , at ) + γQ ( st+1 , at+1 ) ( 10 ) where Q ( st+1 , at+1 ) is the action-value function defined in Eq . 8 to decorrelate q1 ( st+1 , at+1 ) and q2 ( st+1 , at+1 ) , which are both from the frozen target networks . In addition , we want these two qvalue functions as independent as possible . Thus , we introduce corr ( q1 ( st , at ; θq1 ) , q2 ( st , at ; θq1 ) ) , which measures similarity between these two q-value approximators . In the experiment , our method using Eq . 10 can get good results on Halfcheetah , but it did not perform well on other MuJoCo tasks . To stabilize the target value , we take the minimum of Q ( st+1 , at+1 ) and q2 ( st+1 , at+1 ) in Eq . 10 as TD3 Fujimoto et al . ( 2018 ) . Then , it gives the target update of D2Q algorithm below yt = r ( st , at ) + γmin ( Q ( st+1 , at+1 ) , q2 ( st+1 , at+1 ) ) ( 11 ) And the action at+1 is from policy at+1 = π ( st+1 ; θπ ) , which can take a similar policy gradient as in Eq . 6 . Our D2Q leverages the parametric actor-critic algorithm , which maintains two q-value approixmators and a single actor . Thus , the loss in Eq . 9 tries to minimize the three terms below , as corr ( q1 ( st , at ; θ q1 ) , q2 ( st , at ; θ q2 ) ) → 0 q1 ( st , at ; θ q1 ) → yt q2 ( st , at ; θ q2 ) → yt At each time step , we update the pair of critics towards the minimum target value in Eq . 11 , while reducing the correlation between them . The purposes that we introduce control variate q2 ( st , at ) are following : ( 1 ) Since we use q2 ( st , at ) − E ( q2 ( st , at ) ) to model noise , if there is no noise , such that q2 ( st , at ) − E ( q2 ( st , at ) ) = 0 , then we have yt = r ( st , at ) + min ( Qπ ( st , at ) , q2 ( st , at ) ) = r ( st , at ) + min ( q1 ( st , at ) , q2 ( st , at ) ) via Eq . 11 , which is exactly the same as TD3 . ( 2 ) In fact , because of the noise in value estimate , we have q2 ( st , at ) − E ( q2 ( st , at ) ) 6= 0 . The purpose we introduce q2 ( st , at ) is to mitigate overestimate bias in Q-learning . The control variate introduced by q2 ( st , at ) will reduce the variance of Q ( st , at ) to stabilize the learning of value function . Convergence analysis : we claim that our D2Q algorithm is to converge the optimal in the finite MDP settings . There is existed theorem in Jaakkola et al . ( 1994 ) , given the random process { ∆t } taking value in Rn and defined as ∆t+1 ( st , at ) = ( 1− αt ( st , at ) ) ∆t ( st , at ) + αt ( st , at ) Ft ( st , at ) ( 12 ) Then ∆t converges to zero with probability 1 under the following assumptions : 1 . 0 < αt < 1 , ∑ t αt ( x ) =∞ and ∑ t α 2 t ( x ) < ∞ 2 . ||E [ Ft ( x ) |Ft ] ||W ≤ γ||∆t||W + ct with 0 < γ < 1 and ct p→ 0 = 1 3. var [ Ft ( x ) |Ft ] ≤ C ( 1 + ||∆t||2W ) for C > 0 where Ft is a sequence of increasing σ-field such that αt ( st , at ) and ∆t are Ft measurable for t = 1 , 2 , .... Based on the theorem above , we provide sketch of proof which borrows heavily from the proof of convergence of Double Q-learning and TD3 as below : Firstly , the learning rate αt satisfies the condition 1 . Secondly , variance of r ( st , at ) is limit , so condition 3 holds . Finally , we will prove that condition 2 holds below . ∆t+1 ( st , at ) = ( 1− αt ( st , at ) ) ( Q ( st , at ) −Q∗ ( st , at ) ) + αt ( st , at ) ( rt + γmin ( Q ( st , at ) , q2 ( st , at ) ) −Q∗ ( st , at ) ) = ( 1− αt ( st , at ) ) ∆t ( st , at ) + αt ( st , at ) Ft ( st , at ) ( 13 ) where Ft ( st , at ) is defined as : Ft ( st , at ) = rt + γmin ( Q ( st , at ) , q2 ( st , at ) ) −Q∗ ( st , at ) = rt + γmin ( Q ( st , at ) , q2 ( st , at ) ) −Q∗ ( st , at ) + γQ ( st , at ) − γQ ( st , at ) = rt + γQ ( st , at ) −Q∗ ( st , at ) + γmin ( Q ( st , at ) , q2 ( st , at ) ) − γQ ( st , at ) = FQt ( st , at ) + ct ( 14 ) Since we have E [ FQt ( st , at ) |Ft ] ≤ γ||∆t|| under Q-learning , so the condition 2 holds . Then we need to prove ct = min ( Q ( st , at ) , q2 ( st , at ) ) −Q ( st , at ) converges to 0 with probability 1. min ( Q ( st , at ) , q2 ( st , at ) ) −Q ( st , at ) = min ( Q ( st , at ) , q2 ( st , at ) ) − q2 ( st , at ) + q2 ( st , at ) −Q ( st , at ) = min ( Q ( st , at ) − q2 ( st , at ) , 0 ) − ( Q ( st , at ) − q2 ( st , at ) ) = min ( q1 ( st , at ) − q2 ( st , at ) − β ( q2 ( st , at ) − E ( q2 ( st , at ) ) ) , 0 ) + q1 ( st , at ) − q2 ( st , at ) − β ( q2 ( st , at ) − E ( q2 ( st , at ) ) ) ( 15 ) Suppose there exists very small δ1 and δ2 , such that |q1 ( st , at ) − q2 ( st , at ) | ≤ δ1 and |q2 ( st , at ) − E ( q2 ( st , at ) ) | ≤ δ2 , then we have min ( Q ( st , at ) , q2 ( st , at ) ) −Q ( st , at ) ≤2 ( |q1 ( st , at ) − q2 ( st , at ) |+ β|q2 ( st , at ) − E ( q2 ( st , at ) ) | ) =2 ( δ1 + βδ2 ) < 4δ ( 16 ) where δ = max ( δ1 , δ2 ) . Note that ∃δ1 , |q1 ( st , at ) − q2 ( st , at ) | ≤ δ1 holds because ∆t ( q1 , q2 ) = |q1 ( st , at ) − q2 ( st , at ) | converges to zero . According Eq . 9 , both q1 ( st , at ) and q2 ( st , at ) are updated with following qt+1 ( st , at ) = qt ( st , at ) + αt ( st , at ) ( yt − qt ( st , at ) ) ( 17 ) Then we have ∆t+1 ( q1 , q2 ) = ∆t ( q1 , q2 ) − αt ( st , at ) ∆t ( q1 , q2 ) = ( 1 − αt ( st , at ) ) ∆t ( q1 , q2 ) converges to 0 as the learning rate satisfies 0 < αt ( st , at ) < 1 .

Summary: The paper suggests an improvement over double-Q learning by applying the control variates technique to the target Q, in the form of $(q1 - \beta (q2 - E(q2))$ (eqn (8)). To minimize the variance, it suggests minimizing the correlation between $q1$ and $q2$. In addition, it applies the TD3 trick. The resulting algorithm, D2Q, outperforms DDPG and competes with TD3.