Deep Reinforcement Learning With Adaptive Combined Critics

INTRODUCTION . The task of deep reinforcement learning ( DRL ) is to learn good policies by optimizing a discounted cumulative reward through function approximation . Although a variety of control tasks have gained success through DRL Mnih et al . ( 2013 ; 2015 ) ; Van Hasselt et al . ( 2016 ) ; Wang et al . ( 2015 ) ; Schaul et al . ( 2015 ) ; Lillicrap et al . ( 2015 ) ; Mnih et al . ( 2016 ) , there still exists biases caused by function approximation errors , studied in prior work Mannor et al . ( 2007 ) . Overestimation bias is rooted in Q-learning by consistently maximizing a noisy value estimate , and was originally reported to be present in the algorithms , typically , like deep Q-network ( DQN ) Mnih et al . ( 2015 ) which is aimed for discrete control tasks . Since DQN adopts neural networks to approximate a cumulative future reward , it is unavoidable that the noise evolved from function approximation accompanies the whole training . Besides , DQN updates its policy by choosing the action that maximizes the value function , which may bring the inaccurate value approximation that outweighes the true value , i.e. , the overestimation bias . This bias further accumulates at every step via bootstrapping of temporal difference learning , which estimates the value function using the value estimate of a subsequent state . When the overestimation bias is harmful to a certain task , it will cause instability , divergence and suboptimal policy updates . To solve this , Double deep Q-networks ( DDQN ) Van Hasselt et al . ( 2016 ) alter the policy update strategy by taking actions based on a relatively independent target value function instead of maximizing the original Q network . DDQN not only yields more accurate value estimates , but leads to much higher scores on several games Van Hasselt et al . ( 2016 ) . However , due to the randomness of approximation errors , DDQN provides no theoretical bounds for the overestimation , and the unresolved overestimation or induced underestimation makes the performance of DDQN worse than DQN in some cases , reported in Van Hasselt et al . ( 2016 ) . For the case of high-dimensional , continuous action spaces , Deep Deterministic Policy Gradient ( DDPG ) Lillicrap et al . ( 2015 ) provide a model-free , off-policy actor-critic method using deterministic policy gradients . Due to the slow-changing policy in an actor-critic setting , the current and target value estimates will be too dependent to circumvent bias if the solution of DDQN to overestimation is directly under the continuous control setting . Inspired by Double Q-learning Hasselt ( 2010 ) which employ a pair of independently trained critics to achieve a less biased value estimation , the twin delayed deep deterministic policy gradient algorithm ( TD3 ) Fujimoto et al . ( 2018 ) propose a clipped Double Q-learning variant choosing the lower target value as an approximate upper-bound to estimate the true current value , which favors underestimation biases that will not accumulate during training because actions with high value estimation are preferred . However , there are two problems lying in TD3 . First , by taking actions towards the maximization of lower action-value ( Q-value ) at every step , the policy improvement can not be guaranteed , which may cause potential suboptimal policies and instability . Second , using the same target value to update two critic networks will make them less independent . The paper has the following contributions . First , we propose a combined value of two independent critics connected by a weight factor , and use it to update the policy instead of serving as a shared target estimate for the two critics , to avoid losing independence . Second , we propose a sign multiplier which determines whether the updated combined value has increased . The objective function for updating the weight factor is the product of the sign and the combined value evaluated by the updated policy . Third , we present a novel algorithm for continuous control , which can minimize the overestimation bias while providing guarantee for future policy improvement . And theoretical proofs show that the proposed algorithm has the property of asymptotical convergence and expected policy improvement . Fourth , we further apply the proposed algorithm to an unbiased framework to create another algorithm , which can remove the systematic bias due to the probability mismatch between the behavior policy and the target policy existing in off-policy methods . Fifth , extensive evaluations are conducted to compare the proposed algorithms with some baseline algorithms , in terms of computational efficiency , convergence rate and stability . BACKGROUND . The task of reinforcement learning ( RL ) is to learn an optimal policy that maximizes a single return , i.e. , the expected discounted cumulative reward of an episode . During an episode , the agent continually receives a sequence of observations by interacting with the environment before encountering an terminated state or the timeout . The return is calculated as the expected sum of future rewards . DRL combines the neural networks with RL so that the return can be approximated by the parameterized function . In DRL , the agent follows a behavior policy to determine future rewards and states , and store these observations in a memory , which will be randomly sampled over to train the network parameters and update the target Q-values . The target updates can be immediate , delayed or ” soft ” . Generally , the Q-value of DRL is represented as the expected discounted cumulative reward , an estimate function with respect to the state and action , which is given by Qπ ( s , a ) = Epπ ( h|s0 , a0 ) [ ∞∑ t=0 γtr ( st , at ) |s0 = s , a0 = a ] , ( 1 ) where r ( s , a ) is the immediate reward which is usually connected to the state-action pair , ( s , a ) is the value of initial state-action pair , and γ ∈ ( 0 , 1 ) is the discount horizon factor for future rewards . Under the guidance of behavior policy π , pπ ( h|s0 , a0 ) is the joint probability of all state-action pairs during an episode given the initial state-action pair ( s0 , a0 ) . When neural networks are used to approximate Q-values , the update of behavior policy π is closely related to the Q network under the setting of Markov Decision Process ( MDP ) . The Q-value function in ( 1 ) takes the state-action pair ( s , a ) as input and maps it to the Q-value . The foundation for updates of network parameters is the Bellman equation . Most of existing DRL algorithms adopt an independent target Q network to approximate an essential part of the target value , which can be set based on Bellman equation and organized to form the general loss function for DRL as Lillicrap et al . ( 2015 ) L ( ω ) = E ( s , a , r , s′ ) [ ( r + γQ ( s′ , µ ( s′|θ′ ) |ω′ ) −Q ( s , a|ω ) ) 2 ] , ( 2 ) where a is the action drawn from a behavior policy based on s , r and s′ are the immediate reward and next state received by interacting with the MDP environment , respectively . Overall , ( s , a , r , s′ ) is the tuple stored in the replay buffer at every step . Besides , µ ( s′|θ′ ) is the target policy mapping s′ to the next action a′ through a target actor network parameterized by θ′ in deterministic actor-critic methods , instead of taking actions from the replay buffer . Moreover , ω is the parameter of current Q network , which is normally different from the target network parameter ω′ . Once the Q networks are updated , the objective at the current iteration is to optimize the actor parameter θ , which is updated by maximizing the expected return J ( θ ) = Es [ Qπ ( s , µ ( s|θ ) |ω ) ] . In the case of continuous control , θ can be updated by gradient descent∇θJ ( θ ) . ADAPTIVE DELAYED DEEP DETERMINISTIC POLICY GRADIENT . In TD3 , the clipped variant of Double Q-Learning is proposed for actor-critic to reduce the overestimation bias . Instead of using a pair of actors and critics to learn twice , TD3 upper-bounds the less biased value estimate by taking the minimum between two critic networks to offer target estimation for the update of future critics . Inspired by this work , we also adopt two critic networks and their target estimates , which are parameterized by ω , Ω , ω′ and Ω′ , respectively . Since we plan to deal with our problem through DRL , we can substitute sampled minibatches into the loss functions of action-values ( 2 ) , which are given by L ( ω ) = 1 N N∑ n=1 ( rn + γQ ( s ′ n , a ′ n|ω′ ) −Q ( sn , an|ω ) ) 2 , ( 3 ) L ( Ω ) = 1 N N∑ n=1 ( rn + γQ ( s ′ n , a ′ n|Ω′ ) −Q ( sn , an|Ω ) ) 2 , ( 4 ) where a′n = µ ( s ′ n|θ′ ) is the action taken from the target actor network , and ( sn , an , rn , s′n ) is the n-th tuple of minibatches stored in replay buffer . In TD3 , the minimum of two critics is employed to serve as the target estimation value . However , updating two Q networks according to the same target estimate will make them less independent , which will further negatively affect the training efficiency . Besides , it is more reasonable to apply the clipped variant of Double Q-Learning to the procedure of actor update . Inspired by the Lagrange relaxation applied in constrained MDP problems Tessler et al . ( 2018 ) , we propose a dual form of combined value function via a weight factor to determine actions reducing the harmful effect from overestimation . The dual form of combined value function for policy update is given by π∗ = arg min 0≤λ≤1 max π [ ( 1− λ ) Qπ ( s|ω ) + λQπ ( s|Ω ) ] , ∀s ∈ χ \ χ′ . ( 5 ) where Qπ ( s|ω ) and Qπ ( s|Ω ) are two critics following policy π , χ′ be the set of transient states in continuous compact state space χ , and the weight factor λ determines the influence of two Q functions . Specifically , ( 5 ) will reduce to a normal policy evaluation with one critic when λ = 0 or λ = 1 . Generally , ( 5 ) needs to be solved by a two-timescale approach that may result in a saddle point problem , i.e. , on the faster timescale , the policy π or its parameter is solved by ( 5 ) when fixing λ , while on the slower timescale , λ is slowly increased until the overestimation error is minimized without losing the optimal solution to ( 5 ) . Due to the potential non-convexity of the action-value function , the method may lead to a sub-optimal solution , and even cause instability when the convergence is slow . To avoid the potential saddle point problem , we first propose a two-step separation method to solve ( 5 ) for each policy update , which determines the optimal action to restrain overestimation based on the target weight factor λ′ and then use the up-to-date policy to update λ . Second , although the underestimation bias accompanying the minimization operator is far preferable to overestimation and does not explicitly propagate through the policy update , it indeed negatively affect the policy improvement at every iteration and further brings fluctuation on algorithm convergence . The policy improvement means that the optimized objective function in ( 5 ) should steadily increase during training . The insurance of this value improvement lies in the choice of weight factor λ . The process of this separation method is given by J ( θ ) = 1 N N∑ n=1 [ ( 1− λ′ ) Q ( sn , an|ω ) + λ′Q ( sn , an|Ω ) ] , ( 6 ) J ( λ ) = 1 N N∑ n=1 [ ( 1− λ ) Q ( sn , an|ω ) + λQ ( sn , an|Ω ) ] ∗ Sign ( λ ) , ( 7 ) where 0 ≤ λ′ ≤ 1 , an = µ ( sn|θ ) , which means actions in ( 6 ) and ( 7 ) should be taken from the policy of current actor instead of the replay buffer . We provide the guarantee of policy improvement by multiplying the averaged return for λ updating by a sign function , which is given by Sign ( λi ) = I ( 1 N N∑ n=1 [ min λi Q̂ ( sn , an , i|ωi , Ωi , λi ) − Q̂ ( sn , an , i−1|ωi−1 , Ωi−1 , λi−1 ) ] ) , ( 8 ) where Q̂ ( s , a|ω , Ω , λ ) = ( 1− λ ) Q ( s , a|ω ) + λQ ( s , a|Ω ) , ( 9 ) an , i = µ ( sn|θi ) and an , i−1 = µ ( sn|θi−1 ) , which come from actor networks parameterized by current states before and after updating respectively but not from the replay buffer . I ( x ) produces 1 when x is negative , and vice versa . The sign denoted in ( 8 ) is actually the comparison between the minimum updated Q-values in two critics and the old ( before the policy update ) combined value defined in ( 9 ) . Lemma 1 . Denoting the converged values of two critics asQ ( s , a|ω ? ) andQ ( s , a|Ω ? ) , respectively , then the convergence of combined value denoted in ( 9 ) can be ensured by minimizing ( 3 ) and ( 4 ) . Different from updating λ in ( 7 ) , the averaged return for θ updating in ( 6 ) adopts the target weight factor λ′ . Then ( ω′ , Ω′ , θ′ , λ′ ) are updated adopting the ” soft ” target updates Lillicrap et al . ( 2015 ) by ( ω , Ω , θ , λ ) , in the way of ω′i ← τ arg min ωi L ( ωi ) + ( 1− τ ) ω′i−1 , Ω′i ← τ arg min Ωi L ( Ωi ) + ( 1− τ ) Ω′i−1 , θ′i ← τ arg max θi J ( θi ) + ( 1− τ ) θ′i−1 , λ′i ← τ arg min λi J ( λi ) + ( 1− τ ) λ′i−1 , ( 10 ) where τ < 1 is the factor to control the speed of policy updates for the sake of small value error at each iteration , and λ′ updates following θ′ . We organize the above procedures as the adaptive delayed deep deterministic policy gradient ( AD3 ) algorithm , whose pseudocode is described by Algorithm 1 . Theorem 1 . AD3 algorithm asymptotically converges as the iteration i→∞ with properly chosen learning rate . Theorem 2 . AD3 algorithm has the property of asymptotical expected policy improvement . Specifically , when the critics tend to converge , i.e. , ∃K , ∀i ≥ K , ∀ε > 0 , ∣∣∣E ( s , a ) [ Q̂ ( s , a|ωi+1 , Ωi+1 , λi ) − Q̂ ( s , a|ωi , Ωi , λi ) ] ∣∣∣ < ε , ( 11 ) then E ( s , a ) [ Q̂ ( s , a|ωi+1 , Ωi+1 , λi+1 ) ] ≥ E ( s , a ) [ Q̂ ( s , a|ωi , Ωi , λi ) ] . ( 12 ) The proof of Theorem 1 and Theorem 2 can be found in the Appendix . Algorithm 1 AD3 Algorithm 1 : Input : The batch size N , the maximum of updates M , the timeout step T , and the soft update parameter τ . 2 : Initialization : Initialize parameters ( ω , Ω , θ , λ ) ← ( ω0 , Ω0 , θ0 , λ0 ) , ( ω′ , Ω′ , θ′ , λ′ ) ← ( ω′0 , Ω ′ 0 , θ ′ 0 , λ ′ 0 ) randomly ; Initialize replay buffer R , the counter i← 0 . 3 : while i < M do 4 : Reset randomly the initial state s1 . 5 : for t = 1 , T do 6 : Select action at according to the current behavior policy , i.e. , µ ( st|θi ) added by exploration noise ; 7 : Execute actions at , get next states st+1 , and immediate reward rt ; 8 : Store transition ( st , at , rt , st+1 ) in R ; 9 : if R is full then 10 : Randomly and uniformly sample the slot ( si , ai , ri , si+1 ) from R ; 11 : Minimize the Q1 loss function shown in ( 3 ) by gradient decent , and then update ωi ; 12 : Minimize the Q2 loss function shown in ( 4 ) by gradient decent , and then update Ωi ; 13 : Maximize the expected return shown in ( 6 ) by gradient ascent , and then update θi ; 14 : Minimize the product of ( 7 ) and ( 8 ) by gradient ascent , and then update λi ; 15 : Execute the ” soft ” target updates shown in ( 10 ) to update θ′i , ω ′ i , Ω ′ i , and λ ′ i ; 16 : i← i+ 1 ; 17 : end if 18 : end for 19 : end while

Summary: The authors propose a deterministic policy-gradient algorithm that extends the TD3 algorithm (Fujimoto 2018). The main claim is that it reduces overestimation issues in a more effective way. Two Q-critics are maintained with separate parameters, but updated using the same transitions. Then a convex combination of these critics is used in the deterministic policy gradient update. The mixture parameter is learned on a slower time-scale to minimize this convex combination over states (instead of taking the minimum of the 2 critics per batch as in TD3). Another contribution in the paper is the “Unbiased” variant of the algorithm (UAD3), which addresses the off-policy nature of the replay mechanism of the AD3 algorithm described above. My understanding is that this is simply a version of the algorithm that does not use any replay mechanism and samples the state iid from the on-policy distribution, so it isn’t a novel idea in itself.