TEAC: Intergrating Trust Region and Max Entropy Actor Critic for Continuous Control

1 INTRODUCTION . With the use of high-capacity function approximators , such as neural networks , reinforcement learning ( RL ) becomes practical in a wide range of real-world applications , including game playing ( Mnih et al. , 2013 ; Silver et al. , 2016 ) and robotic control ( Levine et al. , 2016 ; Haarnoja et al. , 2018a ) . However , when dealing with the environments with continuous state space or/and continuous action space , most existing deep reinforcement learning ( DRL ) algorithms still suffer from unstable learning processes and are impeded from converging to the optimal policy . The reason for unstable training process can be traced back to the use of greedy or -greedy policy updates in most algorithms . With the greedy update , a small error in value functions may lead to abrupt policy changes during the learning iterations . Unfortunately , the lack of stability in the training process makes the DRL unpractical for many real-world tasks ( Peters et al. , 2010 ; Schulman et al. , 2015 ; Tangkaratt et al. , 2018 ) . Therefore , many policy-based methods have been proposed to improve the stability of policy improvement ( Kakade , 2002 ; Peters & Schaal , 2008 ; Schulman et al. , 2015 ; 2017 ) . Kakade ( 2002 ) proposed a natural policy gradient-based method which inspired the design of trust region policy optimization ( TRPO ) . The trust region , defined by a bound of the Kullback-Leibler ( KL ) divergence between the new and old policy , was formally introduced in Schulman et al . ( 2015 ) to constrain the natural gradient policy changing within the field of trust . An alternative to enforcing a KL divergence constraint is to utilize the clipped surrogate objective , which was used in Proximal Policy Optimization ( PPO ) ( Schulman et al. , 2017 ) to simplify the objective of TRPO while maintaining similar performance . TRPO and PPO have shown significant performance improvement on a set of benchmark tasks . However , these methods are all on-policy methods requiring a large number of on-policy interaction with environment for each gradient step . Besides , these methods focus more on the policy update than exploration , which is not conducive to finding the global optimal policy . The globally optimal behavior is known to be difficult to learn due to sparse rewards and insufficient explorations . In addition to simply maximize the expected reward , maximum entropy RL ( MERL ) ( Ziebart et al. , 2008 ; Toussaint , 2009 ; Haarnoja et al. , 2017 ; Levine , 2018 ) proposes to extend the conventional RL objective with an additional “ entropy bonus ” argument , resulting in the preferences to the policies with higher entropy . The high entropy of the policy explicitly encourages exploration , thus improving the diverse collection of transition pairs , allowing the policy to capture multi-modes of good policies , and preventing from premature convergence to local optima . MERL reforms the reinforcement learning problem into a probabilistic framework to learn energy-based policies to maintain the stochastic property and seek the global optimum . The most representative methods in this category are soft Q-learning ( SQL ) ( Haarnoja et al. , 2017 ) and Soft Actor Critic ( SAC ) ( Haarnoja et al. , 2018b ; c ) . SQL defines a soft Bellman equation and implements it in a practical off-policy algorithm which incorporates the entropy of the policy into the reward to encourage exploration . However , the actor network in SQL is treated as an approximate sampler , and the convergence of the method depends on how well the actor network approximates the true posterior . To address this issue , SAC extends soft Q-learning to actor-critic architecture and proves that a given policy class can converge to the optimal policy in the maximum entropy framework . However , offpolicy DRL is difficult to stabilize in policy improvement procedure ( Sutton & Barto , 1998 ; van Hasselt et al. , 2018 ; Ciosek et al. , 2019 ) which may lead to catastrophic actions , such as ending the episode and preventing further learning . Several models have been proposed to benefit from considering both the trust region constraint and the entropy constraint , such as MOTO ( Akrour et al. , 2016 ) , GAC ( Tangkaratt et al. , 2018 ) , and Trust-PCL ( Nachum et al. , 2018 ) . However , MOTO and GAC can not efficiently deal with highdimensional action space because they rely on second-order computation , and Trust-PCL suffers from algorithm efficiency due to its requirement of trajectory/sub-trajectory samples to satisfy the pathwise soft consistency . Therefore , in this paper , we propose to further explore the research lines of unifying trust region policy-based methods and maximum entropy methods . Specifically , we first transform the RL problem into a primal optimization problem with four additional constraints to 1 ) set an upper bound of KL divergence between the new policy and the old policy to ensure the policy changes are within the region of trust , 2 ) provide a lower bound of the policy entropy to prevent from a premature convergence and encourage sufficient exploration , and 3 ) restrain the optimization problem as a Markov Decision Process ( MDP ) . We then leverage the Lagrangian duality to the optimization problem to redefine the Bellman equation which is used to verify the policy evaluation and guarantee the policy improvement . Thereafter , we propose a practical trust entropy actor critic ( TEAC ) algorithm , which trains the critic by minimizing the refined Bellman error and updates the actor by minimizing KL-divergence loss derived from the closed-form solution to the Lagrangian . The update procedure of the actor involves two dual variables w.r.t . the KL constraint and entropy constraint in the Lagrangian . Based on the Lagrange dual form of the primal optimization problem , we develop gradient-based method to regulate the dual variables regarding the optimization constraints . The key contribution of the paper is a novel off-policy trust-entropy actor-critic ( TEAC ) algorithm for continuous controls in DRL . In comparison with existing methods , the actor of TEAC updates the policy with the information from the old policy and the exponential of the current Q function , and the critic of TEAC updates the Q function with the new Bellman equation . Moreover , we prove that the policy evaluation and policy improvement in trust entropy framework is guaranteed to converge . A detailed comparison with similar work , including MOTO ( Akrour et al. , 2016 ) , GAC ( Tangkaratt et al. , 2018 ) , and Trust-PCL ( Nachum et al. , 2018 ) , is provided in Sec . 4 to explain that TEAC is the most effective and most theoretically complete method . We compare TEAC with 4 state-of-the-art solutions on the tasks in the MuJoCo environment . The results show that TEAC is comparable with the state-of-the-art solutions regarding the stability and sufficient exploration . 2 PRELIMINARIES . A RL problem can be modeled as a standard Markov decision process ( MDP ) , which is represented as a tuple 〈S , A , r , p , p0 , γ〉 . S and A denote the state space and the action space , respectively . p0 ( s ) denotes the initial state distribution . At time t , the agent in state st selects an action at according to the policy π ( a|s ) , in which the performance of the state-action pair is quantified by the reward function r ( st , at ) and the next state of the agent is decided by the transition probability as st+1 ∼ p ( st+1|st , at ) . The goal of the agent is to find the optimal policy π ( a|s ) to maximize the expected reward Es0 , a0 , ... [ ∑∞ t=0 γ tr ( st , at ) ] , where s0 ∼ p0 ( s ) and st+1 ∼ p ( st+1|st , at ) . γ is a discount factor ( 0 < γ < 1 ) ) which quantifies how much importance we give for future rewards . The state-action value function Qπ ( st , at ) and the value function V π ( st ) are then defined as : Qπ ( st , at ) = Est+1 , at+1 , ... [ ∞∑ l=0 γlr ( st+l , at+l ) ] , V π ( st ) = Eat , st+1 , ... [ ∞∑ l=0 γlr ( st+l , at+l ) ] . For the continuous environments , which is the focus of this paper , S andA denote finite dimensional real valued vector spaces , s denotes the real-valued state vector , and a denotes the real-valued action vector . The expected reward can be defined as : J ( π ) = E ( s , a ) ∼ρπ ( s , a ) [ Q π ( s , a ) ] = Eρπ ( s ) π ( a|s ) [ Q π ( s , a ) ] , ( 1 ) where ρπ ( s ) and ρπ ( s , a ) denote the ( discounted ) state and ( discounted ) state-action marginals of the trajectory distribution induced by a policy π ( a|s ) . 1 3 OUR METHOD . This section explains the details and features of the TEAC framework with the focus on the mathematical deductions and proofs of the guaranteed policy improvement and convergence in an actorcritic architecture . 3.1 PRIMAL AND DUAL OPTIMIZATION PROBLEM . To stabilize the training process and steer the exploration , in addition to simply maximizing the expected reward with ( - ) greedy policy updates , we propose to 1 ) confine the KL-divergence between neighboring policies in the training procedure to avoid large-step policy updates , and 2 ) favor a stochastic policy with relatively larger entropy to avoid premature convergence due to insufficient exploration . Therefore , we define the RL problem as a primal optimization problem with additional constraints , given as : maxπ Eρπ ( s ) π ( a|s ) [ Q̂ ( s , a ) ] , subject to Eρπ ( s ) [ KL ( π ( ·|s ) ‖πold ( ·|s ) ) ] ≤ τ , Eρπ ( s ) [ H ( π ( ·|s ) ) ] ≥ η , Eρπ ( s ) ∫ π ( a|s ) da = 1 , Eρπ ( s ) π ( a|s ) p ( s′|s , a ) V̂ ( s′ ) = Eρπ ( s′ ) V̂ ( s′ ) , ( 2 ) where Q̂ ( s , a ) is a critic estimating the state-action value function whose parameter is learned such that Q̂ ( s , a ) ≈ Qπ ( s , a ) , π ( ·|s ) is the policy distribution to be learned , πold ( ·|s ) is the prior policy distribution , and V̂ ( s′ ) is a state feature function estimating the state value function of the next state . The term KL ( π ( ·|s ) ‖πold ( ·|s ) ) = Eπ ( a|s ) [ log π ( a|s ) − log πold ( a|s ) ] confines the KL-divergence between the distributions of the new and old policies . The third constraint ensures that the stateaction marginal of the trajectory distribution is a proper probability density function . As the state marginal of the trajectory distribution needs to comply with the policy π ( a|s ) and the system dynamics p ( s′|s , a ) , i.e. , ρπ ( s ) π ( a|s ) p ( s′|s , a ) = ρπ ( s′ ) , meanwhile the direct matching of the state probabilities is not feasible in continuous state spaces , the use of V̂ ( s′ ) in the fourth constraint which can be also considered as state features , helps to focus on matching the feature averages . These last two constraints formally restrain the optimization problem within a MDP framework . The objective is to maximize the expected reward of a policy while ensuring it satisfies the lower bound of entropy and upper bound of distance from the previous policy . The constraint of KLdivergence term helps to avoid the abrupt difference between the new and old policies , while the constraint of the entropy term helps to promote the policy exploration . The entropy constraint is crucial in our optimization problem for two reasons : 1 ) Prior studies show that the use of KL-bound leads to a rapid decrease of the entropy , thus bounding the entropy helps to lower the risk of premature convergence induced by the KL-bound ; 2 ) Each iteration of policy update will modify the critic Q̂ ( s , a ) and the state distribution ρπ ( s ) , thus changing the optimization 1Following Sutton et al . ( 2000 ) , we use ρπ in the paper to implicate that ρπ is the stationary distribution of states under π and independent of s0 for all policies . landscape of the policy parameters . The entropy constraint ensures the exploration in the action space in case of evolving optimization landscapes . The Lagrangian of this optimization problem is denoted as : L ( π , α , β , λ , ν ) =Eρ ( s ) π ( a|s ) [ Q̂ ( s , a ) ] + α ( τ − Eρ ( s ) [ KL ( π ( ·|s ) ‖πold ( ·|s ) ) ] ) + β ( Eρ ( s ) [ H ( π ( ·|s ) ) ] − η ) + λ ( Eρ ( s ) ∫ π ( a|s ) da− 1 ) + ν ( Eρ ( s ) π ( a|s ) p ( s′|s , a ) V̂ ( s′ ) − Eρ ( s′ ) V̂ ( s′ ) ) , ( 3 ) where α , β , λ , ν are the dual variables , and for the sake of brevity , we use ρ ( s ) to represent ρπ ( s ) . Eq . 3 is a super set of trust region and maximum entropy methods . That is , β = 0 leads to an equivalent objective function as the standard trust region , while α = 0 , which indicates that the KL-divergence bound is not active , leads to a maximum entropy RL objective that SAC tries to solve . Take derivative of L w.r.t . π and set the derivative to zero : ∂πL =Eρ ( s ) [ ∫ ( Q̂ ( s , a ) − ( α+ β ) log π ( a|s ) + α log πold ( a|s ) − νV̂ ( s ) + Ep ( s′|s , a ) [ νV̂ ( s′ ) ] ) da ] − ( α+ β + λ ) =Q̂ ( s , a ) − ( α+ β ) log π ( a|s ) + α log πold ( a|s ) − νV̂ ( s ) + Ep ( s′|s , a ) [ νV̂ ( s′ ) ] − ( α+ β + λ ) =0 . ( 4 ) Continuous problem domains require a practical approximation to the policy update function . We use neural networks as function approximators to parameterize the policy and Q function . Specifically , the Q function , known as critic , is modeled as expressive neural networks Qφ ( s , a ) , and we follow Lillicrap et al . ( 2016 ) to build a target critic network Qφ̄ which mitigates the challenge of overestimation . Meanwhile , the policy , known as actor , is parameterized by πθ ( ·|s ) as a Gaussian with mean and covariance given by neural networks , and we also build up another neural network πθ̂ ( ·|s ) with the same architecture as πθ to enable us to facilitate policy learning by leveraging the “ old ” policy within our framework .

Summary: This paper proposes Trust Entropy Actor Critic (TEAC), a novel algorithm for reinforcement learning (RL) combining the idea of TRPO/PPO and max-entropy RL, together with the corresponding critic, actor and dual updates. The high level idea is that trust region methods ensure stability by constraining the KL divergence from the previous policy, while entropy regularization encourages exploration, and hence combining the two may achieve the best of both worlds and obtain a good trade-off between stability and exploration. To achieve this goal, the authors propose to augment the original trust-region subproblem in TRPO with an additional constraint on the lower bound of the policy entropy (together with two other trivial constraints corresponding to the validity of the policy in the MDP framework). Then by forming the Lagrangian function and setting the gradient to zero, the authors obtain both the critic (value) and actor (policy) updates with different choices of dual variables (corresponding to the two trivial constraints), together with the dual updates. Numerical experiments compared to some popular baseline RL algorithms are also reported to demonstrate the improvement of TEAC compared to the existing works.