TEAC: Intergrating Trust Region and Max Entropy Actor Critic for Continuous Control
Authors: Hongyu Zang
Xin Li
Li Zhang
Peiyao Zhao
Mingzhong Wang
Venue:  (2021)
Keywords: Reinforcement Learning
Trust region methods
Maximum Entropy Reinforcement Learning
Deep Reinforcement Learning

Abstract: Trust region methods and maximum entropy methods are two state-of-the-art branches used in reinforcement learning (RL) for the benefits of stability and exploration in continuous environments, respectively. This paper proposes to integrate both branches in a unified framework, thus benefiting from both sides. We first transform the original RL objective to a constraint optimization problem and then proposes trust entropy actor-critic (TEAC), an off-policy algorithm to learn stable and sufficiently explored policies for continuous states and actions. TEAC trains the critic by minimizing the refined Bellman error and updates the actor by minimizing KL-divergence loss derived from the closed-form solution to the Lagrangian. 
We prove that the policy evaluation and policy improvement in TEAC is guaranteed to converge.
We compare TEAC with 4 state-of-the-art solutions on 6 tasks in the MuJoCo environment. The results show that TEAC outperforms state-of-the-art solutions in terms of efficiency and effectiveness.


Under review as a conference paper at ICLR 2021
TEAC: INTEGRATING TRUST REGION AND MAX ENTROPY ACTOR CRITIC FOR CONTINUOUS CONTROL
Anonymous authors Paper under double-blind review
ABSTRACT
Trust region methods and maximum entropy methods are two state-of-the-art branches used in reinforcement learning (RL) for the benefits of stability and ex- ploration in continuous environments, respectively. This paper proposes to inte- grate both branches in unified framework, thus benefiting from both sides. We first transform the original RL objective to a constraint optimization problem and then proposes trust entropy actor-critic (TEAC), an off- off-policy algorithm to learn stable and sufficiently explored policies for continuous states and actions. TEAC trains the critic by minimizing the refined Bellman error and updates the actor by minimizing KI Kl.-divergence loss derived from the closed-form solution to the La- grangian. We prove that the policy evaluation and policy improvement in TEAC is guaranteed to converge. We compare TEAC with 4 state-of-the-art solutions on 6 tasks in the MuJoCo environment. The results show that TEAC with opti- mized parameters achieves similar performance in half of the tasks and notably improvement in the others in terms of efficiency and effectiveness.
INTRODUCTION
With the use of high-capacity function approximators, such as neural networks, reinforcement learn- ing (RL) becomes practical in wide range of real-world applications, including game playing (Mnih) et al. 2013: Silver et al.] 2016) and robotic control (Levine et al. [2016] Haarnoja et al.) 201 8a). However, when dealing with the environments with continuous state space or/and continuous ac- tion space, most existing deep reinforcement learning (DRL) algorithms still suffer from unstable learning processes and are impeded from converging to the optimal policy.
The reason for unstable training process can be traced back to the use of greedy or c-greedy policy updates in most algorithms. With the greedy update, small error in value functions may lead to abrupt policy changes during the learning iterations. Unfortunately. the lack of stability in the training process makes the DRL unpractical for many real- woal-world tasks (Peters et al. 2010 (Schulman 2015 Tangkaratt et al. 2018). Therefore, many policy-based based methods have been proposed to improve the stability of policy improvement (Kakade) [2002] Peters & Schaal 2008 Schulman al. 2015 2017). Kakade (2002) proposed natural policy gradient-based method which inspired the design of trust region policy optimization (TRPÓ). The trust region, defined by bound of the Kullback-Leibler (KL) divergence between the new and old policy, was formally introduced in Schulman et al. (2015) to constrain the natural gradient policy changing within the field of trust. An alternative to enforcing a KL divergence constraint utilize the clipped surrogate objective, which was used in Proximal Policy Optimization (PPO) (Schulman et al. 2017) to simplify the objective of TRPO while maintaining similar performance. TRPO and PPO have shown significant performance improvement on set of benchmark tasks. However, these methods are all on-policy methods requiring large number of on-policy interaction with environment for each gradient step. Besides, these methods focus more on the policy update than exploration, which is not conducive to finding the global optimal policy.
eet al.
The globally optimal behavior is known to be difficult to learn due to sparse rewards and insuf- ficient explorations. In addition to simply maximize the expected reward, maximum entropy RL (MERL) (Ziebart al.) 2008 Toussaint 2009; Haarnoja et al. 2017 Levine 2018) proposes to extend the conventional RL objective with an additional "entropy bonus argument, resulting in the preferences to the policies with higher entropy. The entropy of the policy explicitly encourages

---

Under review as a conference paper at ICLR 2021
exploration, thus improving the diverse collection of transition pairs, allowing the policy to capture multi-modes of good policies, and preventing from premature convergence to local optima. MERL reforms the reinforcement learning problem into a probabilistic framework to learn emeryy-bascl policies to maintain the stochastic property and seek the global optimum. The most representative methods in this category are soft Q-learning (SQL.) (Haarnoja al.] 2017) and Soft Actor Critic (SAC) (Haarnoja et al. 2018b SQL defines a soft Bellman equation and implements it in a prac- tical off-policy algorithm which incorporates the entropy of the policy into the reward to encourage exploration. However, the actor network in SQL is treated as an approximate sampler, and the con- vergence of the method depends on how well the actor network approximates the true posterior. To address this issue, SAC extends soft Q-learning to actor-critic architecture and proves that a given policy class can converge to the optimal policy in the maximum entropy framework. However, off- policy DRL is difficult to stabilize in policy improvement procedure (Sutton & Barto [998 van Hasselt et al. 2018: Ciosek et al. 2019) which may lead to catastrophic actions, such as ending the episode and preventing further learning.
Several models have been proposed to benefit from considering both the trust region constraint and the entropy constraint, such as MOTO (Akrour al.) 2016) GAC (Tangkaratt et al.] [018)) and Trust-PCL (Nachum et al. 2018). However, MOTO and GAC cannot efficiently deal with high- dimensional action space because they rely second-order computation, and Trust-PCL. suffers from algorithm efficiency due to its requirement of trajectory/sub--rjjeccooy samples to satisfy the pathwise soft consistency.
Therefore, in this paper, we propose to further explore the research lines of unifying trust region policy-based methods and maximum entropy methods. Specifically, we first transform the RL prob- lem into a primal optimization problem with four additional constraints to 1) set an upper bound of KL divergence between the new policy and the old policy to ensure the policy changes are within the region of trust, 2) provide a lower bound of the policy entropy to prevent from a premature con- vergence and encourage sufficient exploration, and 3) restrain the optimization problem as a Markov Decision Process (MDP). We then leverage the Lagrangian duality to the optimization problem to redefine the Bellman equation which is used to verify the policy evaluation and guarantee the pol- icy improvement. Thereafter, we propose a practical trust entropy actor critic (TEAC) algorithm, which trains the critic by minimizing the refined Bellman error and updates the actor by minimizing KL-divergence loss derived from the closed-fome solution to the Lagrangian. The update proce- dure of the actor involves two dual variables w.r.t. the KL. constraint and entropy constraint in the Lagrangian. Based on the Lagrange dual form of the primal optimization problem, we develop gradient-based method to regulate the dual variables regarding the optimization constraints.
The key contribution of the paper is novel off-policy trust-entropy actor-critic (TEAC) algorithm for continuous controls in DRL. In comparison with existing methods, the actor of TEAC updates the policy with the information from the old policy and the exponential of the current Q function, and the critic of TEAC updates the Q function with the new Bellman equation. Moreover, we prove that the policy evaluation and policy improvement in trust entropy framework is guaranteed to converge. A detailed comparison with similar work, including MOTO (Akrour et al. (016)) GAC (Tangkarati jet al. 2018), and PCLL (Nachum et al. 2018), is provided in Sec. 44to explain that TEAC is the most effective and most theoretically complete method. We compare TEAC with 4 state-of-the-art solutions on the tasks in the MuJoCo environment. The results show that TEAC is comparable with the state-of-the-art solutions regarding the stability and sufficient exploration.
2 PRELIMINARIES
A RL problem can be modeled as a standard Markov decision process (MDP), which is represented as a tuple (S, A, P. 7). S and A denote the state space and the action space, respectively. po(s) denotes the initial state distribution. At time e, the agent in state 8, selects an action a, according to the policy (als), in which the performance of the state-action pair is quantified by the reward function r(st, ax) and the next state of the agent is decided by the transition probability as maximize the expected reward Es....... 15:0 7'r (s,, az)], where Sp ~ (s( and 8,+1 ~ p(8:-11/ss ||, az). y is a discount factor (0 <7 7- < 1)) which quantifies how much importance we give for future rewards.
S141 p($ St, a.). The goal of the agent is to find the optimal policy (als))
2

---

Under review as a conference paper at ICLR 2021
The state-action value function (st, and the value function V (s() are then defined as:
DO
Σ Ar (SL-1, V (8t) Bag
Σ Ar (44tt
Q" (St, 1
For the continuous environments, which the focus of this paper, S and A denote finite dimensional real valued vector spaces, denotes the real-valued state vector, and denotes the real-valued action vector. The expected reward can be defined as:
( (ss.a))) (8" a)] (p)
(Q" a)l
(1)
where Pa (s) and Pn (s, a) denote the (discounted) state and (discounted) state-action marginals of the trajectory distribution induced by policy π (aass)
OUR METHOD
This section explains the details and features of the TEAC framework with the focus on the mathe- matical deductions and proof' of the guaranteed policy improvement and convergence in an actor- critic architecture.
3.1 PRIMAL AND DUAL OPTIMIZATION PROBLEM
To stabilize the training process and steer the exploration, in addition simply maximizing the expected reward with (e-) greedy policy updates, we propose to 1) confine the KL-divergence be- tween neighboring policies in the training procedure avoid large-step policy updates, and favor a stochastic policy with relatively larger entropy to avoid premature convergence due to insufficient exploration. Therefore, we define the RL problem as a primal optimization problem with additional constraints, given as:
E (a)x((aa)) [Q(s, a)); E (s) [KL (T(-|s) (s) [-aa( S,,
max, subject to
E Ep)( [[(T() 8) s)] 2 17,
Ep,(,) [(a da 1,
[ex (s') = (x)) V(8),,
=
where Q(s,, a) is a critic estimating the state-action value function whose parameter is learned such that Q(s, a) = (s, a), (-|ss is the policy distribution to be learned, (- (s) is the prior policy distribution, and V (s') is a state feature function estimating the state value function of the next state. The term KL (-(s) o [+s))
Ea a) [log π (aass log [aua(a confines the KL--dvergence between the distributions of the new and old policies. The third constraint ensures that the state- action marginal of the trajectory distribution
proper probability density function. As the state marginal of the trajectory distribution needs to comply with the policy ={ass) and the system dy- namics p(s' 8, a), i.e., Px (a)==(a) 8 s))((" |s, a) = Pa (s'), meanwhile the direct matching of the state probabilities is not feasible in continuous state spaces, the use of (s'') in the fourth constraint which can be also considered state features, helps to focus on matching the feature averages. These last two constraints formally restrain the optimization problem within a MDP framework.
The objective is maximize the expected reward of policy while ensuring it satisfies the lower bound of entropy and upper bound of distance from the previous policy. The constraint of KL- divergence term helps avoid the abrupt difference between the new and old policies, while the constraint of the entropy term helps to promote the policy exploration.
The entropy constraint is crucial in our optimization problem for two reasons: 1) Prior studies show that the use of KL-bound leads to a rapid decrease of the entropy, thus bounding the entropy helps to lower the risk of premature convergence induced by the KL-bound; 2) Each iteration of policy update will modify the critic Q(s, ((s, a. and the state distribution Ps (s), thus changing the optimization
Following Sutton al. 2000), we use P= in the paper implicate that Pa is the stationary distribution of states under π and independent of 8q for all policies.
3

---

Under review as a conference paper at ICLR 2021
landscape of the policy parameters. The entropy constraint ensures the exploration in the action space in case evolving optimization landscapes.
The Lagrangian of this optimization problem is denoted as:
a, - -B (ppt))((aas) a)] + a (r Ep(s) [KL (T(-1s]][=aaa((11))
+ 8 (B (Bp(s) (H(( ())] - 11) + A Bp(ss) / x(als)da -
(3)
( V (s')
V(s'),
where a, 8, A. V are the dual variables, and for the sake of brevity, we use p(s) to represent Pr (s). Eq. [ is super set of trust region and maximum entropy methods. That is, 0 leads to equivalent objective function as the standard trust region, while a = 0, which indicates that the Kl-divergence bound is not active, leads maximum entropy RL objective that SAC tries to
solve.
Take derivative of w.r.t and set the derivative zero:
/ (Qls. (a (a ((als) - (ol (V())+
O, -Bp(s)
Ep(orl (oViiaa)lVV(()) (5')) (
3
(4)
-Q(s,a) a) - la B) log lols( log (als) vV(s) )p (vV
(a ()
=0.
Continuous problem domains require a practical approximation to the policy update function. We neural networks function approximators parameterize the policy and function. Specifi- cally, the Q function, known as critic, is modeled as expressive neural networks Qa (s, a). and we follow Lillicrp al. (2016) build a target critic network Q3 which mitigates the challenge of overestimation. Meanwhile, the policy, known actor, is parameterized by =a(-1s) with mean and covariance given by neural networks, and we also build up another neural network Ta(+ with the same architecture as Te to enable facilitate policy learning by leveraging the "old" policy within our framework.
Gaussian
3.2 CRITIC UPDATE
Given the fact that we sample actions from the actor network as parameterized Gaussian distribution and the value function should satisfy Bellman equation?]
(s) = Q(s, a) (a + 3) log (a)s) (s) + a log Tou(a))) + (v(r)))
A) (5)
The last constant term in Eq can be ignored as it does not affect the Bellman iteration when neural networks used approximate the value function. Therefore, the Bellman equation can be redefined in our framework.
According Eq 5 we could compute the value of a fixed policy æ. Starting from any function Q : x A + R, we define our modified Bellman backup operator.
Definition 3.1 Bellman Equation. modified Bellman backup operator T is defined
T=Q(s, a) 2 + ( (Vs')]
(6)
where
V(8) Ex(ale) [Q(s, a) - (ax + 3) log 7 (a 8) + a log Tous(a 8)]
is the trust entropy state-value function in our framework.
'e could utilize any form of state features ((a)). Thus, vV(')) can he seen another form of state features. Therefore, he arbitrary
4

---

Under review as a conference paper at ICLR 2021
In the sequel, Q(s, a) stands for the state-action value function obtained by iteratively applying the modified Bellman backup operator in Eq 6 and 7 which is the trust entropy Q-value in our framework. Meanwhile, the policy evaluation can be verified accordingly.
Lemma 1 (Trust Entropy Policy Evaluation). Lei eet+===> Q* the sequence Q6 will converge to the trust entropy Q-value of as k + OC when considering a mapping Q" : x A + R with |A| < oo and the Bellman backup operator T
Proof. See Appendix A.1
The learning of Q function can utilize off-po methods to acquire high sample efficiency. Hence, the parameters can be trained by minimizing the squared Bellman error. given as:
1 [/2 (Qo (s) -y)²
Lo(d) -eo
(8)
where D is the replay buffer, Qo(s, a) represents the Q network (also known critic network) which is parameterized by Φ, and y (s) =ZE [V (s') where Va denotes the value function obtained from the target critic network Q with Eq. The inclusion of the target critic network helps to stabilize the training. As suggested in Lillierap et al. (2016), ø is updated via f- KΦ + (1 z) where 0 S K K 1. We set A = 0.005 in the experiments. The expected squared Bellman error computed with samples drawn from the replay buffer using mini-batch. Thus, the approximate gradient of the squared Bellman error Lo w.r.t. Φ is:
(6)
(ol
=v (a, 8) (Qs (8, a) (r(s, a) + 7 (s',a') (a 3) log no (a + a log Ty (als'))),
(9)
where 0 and 0 are parameters of current policy and old policy respectively, and a' is sampled from current policy To given s'.
3.3 ACTOR UPDATE
Setting = 0 in Eq in fact does not change the optimization problem. Therefore, a closed-form solution regarding the policy is given as:
((s,a) a 8
a 3 + + 3
x (a|s) Tata (ala(a)s) exp
exp
(10)
((())))
(als( exp
a + 8
where exp - a+
is the normalization term of T(aas) (The detailed derivation is provided
in A.2).. It should be noted that MORE (Daniel et al., 2016). MOTO (Akrour et al. 2016), GAC Tangkaratt al. 2018), and Trust-PCL (Nachum al. (0018) can also been viewed prior work stemming from Eq 10 However, it is infeasible to use Eq 10 to directly update the policy given the fact that we cannot guarantee that the resulting policy remains in the same policy class when weigh- ing the old policy with the exponential of Q function without any assumption. Different strategies have been applied in the prior work address this issue, and the detailed discussion provided in Sec. 4
To improve the tractability of policies, as Haarnoja al. (2018c). we require the policy is selected from set of policies E II, which is parameterized Gaussian distribution family. This is guaran- teed by the use the Kullback-Leibler divergence to ensure the improved policy locates in the same
policy Since the normalization term exp the gradient of the new policy, it can be ignored. Therefore, the policy is updated by
a+ a+
intractable and does not contribute to
[((s. a) a + 3
C (0) = EEND DKL Th (a|s) || (T((a()) (a((al))=+ ot exp
(11)
5

---

Under review as a conference paper at ICLR 2021
where D is the replay buffer, Ty represents the parameterizze policy, "a represents the parameterized old policy, and a in Q is sampled from the current policy тę- In practice, the old policy network equals to the policy network in the last iteration. Therefore, could leverage another actor network keep the old policy by copying 0 to 0 after computing the loss function of the policy and before the back propagation in each iteration (The detailed algorithm is provided in Appendix C.. With the assumption of policy being Gaussian, the policy improvement can be guaranteed in our framework.
Lemma 2 (Trust Entropy Policy Improvement). Given policy and an old policy a, define new policy
(0*(3, .) a a+
* *(als) (aals) exp
vss.
(12)
If Q is bounded, J [(aas) exp ( da bounded for any 8 (for all a, and R), and the policies are the parameterized Gaussian networks. Then we can obtain Q (s. 2 (= Vs,0 a.
Proof. See Appendix A.3
In other words, in the policy improvement, tial of the Q function induced by the current policy to derive the policy of next iteration. Because the function is non-linear function approximation parameterized by neural networks and can be differentiated, the reparameterization trick a [e( 8), where & is an input noise vector sampled from standard normal distribution, can be applied. Then, the approximate gradient of C w.r.t. is given as:
the information from the old policy and exponen-
(0) = Va(a + B) log (To(als)) aa
(13)
3.4 DUAL VARIABLES UPDATE
This section explains the updates of dual variables (a and (3) throughout the entire framework.
The dual function, which is derived by substituting ((]s)) in the Lagrangian (Eq. 3) with its form in TO is given as:
a + 3 + A
g(a ar -3n (a + B) Ep(s)
(14)
As exp (++8+) o+B
is the normalization term of π (als), the dual function can be represented as:
g(a, B) Bn + Ep(()) [a log Toma (a|s) + Q(s, a) (a 3) log (a]s))
(15)
The approximate gradient of g(a, () w.r.t. a and B3 are:
Vag(a) (a) r - log To(a)s) + log (a(a)))
(16)
v (9 -1 77 log no (a|s).
(17)
which enable us to find the "proper" and / with the gradient descent method, satisfying the KL and entropy constraints in 2 The dual variable updates, along with the trust entropy Q function updates (Sec. (3.2) and trust entropy policy updates (Sec. (33).3.3). constitute the main components of our framework.
4 CONNECTION WITH PREVIOUS WORK
The most related methods to our work are MOTO (Akrour et al. 2016) GAC (Tangkaratt et al.) 2018), and Trust-PCL. (Nachum al. 2018) they also consider both the trust region constraint and the entropy constraint.
MORE DDnniel al. 2016) considers the two constraints in the domain of stochastic search opti- mization. MOTO Akrour et al. 2016) extends MORE to the sequential decision making domain. In MOTO, function is estimated by using quadratic surrogate function of the state and action space,

---

Under review as a conference paper at ICLR 2021
and the policy of log-linear Gaussian form is updated according to the KL-divergence bounding constraint and a variable lower bound of entropy determined by the policy of each iteration.
GAC (Tangkaratt et al. 2018) further extends MOTO by 1) approximating the Q function with truncated Taylor series expansion and parameterizing them with a deep neural network, and 2) learn- ing log-nonlinear Gaussian form policies. In some sense, MOTO, GAC, and ours be viewed as solving the optimization problem with the same constraints as stated in Eq 2 After leveraging the Lagrangian of the optimization problem, the corresponding closed form solution for the policy up- dating is shown in Eq [O] However, Eq [T] indicates that the new policy is derived by weighing the old policy and the exponential of Q function (see the R.H.S of Iq which may deviate the updated policy from the expected policy distribution class. Consequently, the KL constraint will be no longer preserved (Akrour et al., 2018). These methods differ from each other by using different strategies to circumvent the issue. MOTO utilizes quadratic Q function and assumes the policy is of log-linear Gaussian form. Consequently, MOTO can update the policy way. GAC adopts the similar strategy as MOTO to learn oonamarrmeteri Gaussian actor, and then uses this actor to guide a parameterized actor with supervised learning. However, hard for MOTO and GAC to deal with nigh-dimensional action space because they rely on second-order computation. In comparison, we redefine the Bellman equation and guarantee the policy improve- ment by updating policies with [ Therefore, our method resolves the challenge simply with more general assumption of Gaussian policy class.
non-parameterized
Moreover, when dealing with the dual function
Q(8 a) a + B
g(a, 3) = at - Bq B
(a + B)
log Motal Mota( (a)u(a))))))))) म exp
da
(18)
where the integral term is intractable, MOTO and GAC reply on complex second-order computation to make it tractable. contrast, we resolve the challenge by leveraging the policy to transform the dual function into simpler form which can still be optimized in first-order computation, e.g., stochastic gradient descent method, with the policy improvement guaranteed.
Trust-PCL (Nachum et al. 2018 addresses the challenge with different perspective by integrating path consistency learning (PCL) (Nachum et al., (017)). which is developed in the maximum en- tropy framework, and trust region policy optimization method. PCL., which is the base algorithm of trust-PCL, suggests that the optimal policy and state values should satisfy pathwise soft consistency property along any sampled trajectory, thus allowing the use of off-policy data. Consequently, the single-step temporal consistency of state-value function in Trust-PCL is
V (*x) = = [r: - (T x) log (az |s)) + log ā (ar+s (8++)) + 7V* (8441),
(19)
which is similar to our state-value function. However, our method differs from Trust-PCL. in 3 ways: 1) Trust-PCL. focuses on updating the state-value function while our method focuses on updating the Q function. 2) Trust-PCL. updates the policy directly with the temporal consistency squared error while we use Eq II 3) As each update iteration in Trust-PCL requires trajectory/ssub-rrjcccoooo sam- ples to satisfy the pathwise soft consistency, it significantly compromises the algorithm efficiency. In comparison, our method requires only state-action pairs for each update iteration and is capable of finding (sub-)oppl suub)ooptimal value for every dual variable in each update iteration.
should be noted that proximal policy optimization (PPO) (Schulman et al. 2017) can also achieve trust region constraint and encourage unpredictably actions by adding entropy bonus its loss function. However, the entropy bonus PPO is one-off effect which only considers the current state but no future states of the agent. In comparison, our method can be considered as maximum long-term entropy with constraint policy in trust region.
5 EXPERIMENTS
We experimented to investigate the following questions: (1) In comparison with state-of-the-ant art algorithms, does TEAC have better performance in terms of sample efficiency and computational efficiency? (2) How should we choose hyperparameters т and n and how can these two variables affect the performance?
7

---

Under review as a conference paper at ICLR 2021
14002
7032
10.00
10003
2500
8930
4302
6003
3303
1000
2202
2302
0.5
1.0
1.5
2.0
2.5
3.0
0.5
5.0
2.5
2.0
3.5
20
35
3.0
2.5
2.0
25
Lamicalaao
(a) HalfCheetah--3
(b) Hopper-v3
(c) Humanoid-v3
1000
140
5030
1990
120
50.00
48.0
4030
3030
2030
40
1000
2.2
Brnsstope
105
385
105
(d) Walker2d-y3
(c) Swimmer-v3
(f) Ant-v3
Figure/ 1: Performance comparisons on six MuloCo tasks trained for million timesteps. The horizontal axis indicates number of environment steps. The vertical axis indicates the average return. We trained three different instances of each algorithm with different random seeds, with evaluation every 4,000 environment steps. The solid lines represent
each instance performing the mean and the shaded regions mark the minimum and maximum returns over the three trials. We set 1) as the negative of action space dimension of the task, and set 0.005 for all tasks.
5.1 SETUP
We experimented the continuous control tasks (HalfCheettah-v3, Hopper-v3, Humanoid-v3, Walker2d-v3, Swimmer-v3, Ant-v3) available from the MuJoCo environment (Todorov et 2012).
We compared our method TEAC with 1) proximal policy optimization (PPO) (Schulman et al. 2017], stable and effective on-policy policy gradient algorithm; 2) SAC (Haarnoja al. (018c), the state-of-the-ar off-policy algorithm for learning maximum entropy policies whose temperature is adjusted automatically: 3) Trust-PCL (Nachum et al. 2018), an off-policy method optimizing maximum entropy RL objective with trust region; and 4) GAC (Tangkaratt al. 2018), an off- policy method utilizing second-order information of critic. For Trust-PCL and GAC, we used their original implementation provided by their 4 For PPO and SAC, used the implemen- tation publicly provided by OpenAl and adapted SAC to its automatically adjusting version in Haarnoja et al. (201c)). For convenience,
developed our algorithm based spinningup version of SAC The podu of our method is provided in Appendix C and the source code is available ulattps://gitt http:/ttttuu.c..cc.eeeeeeee httpa//cmmcccc///22eeeee....
TEAC requires to specify hyperparameters т. which represents the desired maximum KL- divergence, and v,, which specifies the desired minimum entropy, before training. As the require- ments of stability and exploration vary in different tasks, we set as the negative of action space dimension of the task, and set 0.005 for all tasks. The settings of effective hyperparameters are provided in Appendix D
"Trust-PCL code: http://hhh...ooomm///o//oo////// http://wwwoooooeee/////////reann trec/masterrhescarcchpll.
'GAC code: https://github.cm www//ootttgguidd-aaeeoooooooeeee
*Duc to the second-order computation complexity, we only finished testing GAC on HalfCheetah-và and Hopper-v3 at the time of paper submission. The experimental results show that we achieve better perfor- mance than GAC in much shorter running time. We will completc the comparison before the rebuttal begins.
"OpenAl spinningup code: http:///gghubbboommmoooo///iiign hwtp://rw.tcciiiiiiiooooooooeeer
8

---

Under review as a conference paper at ICLR 2021
5.2 RESULTS
Fig. illlutrrate the training curve for each algorithm. In general, when 0.005, TEAC has simi- lar performance as SAC in simpler tasks which have lower dimension of actions, such Hopper-v3, Walker2d-v3, and Ant-v3. However, for complex tasks with higher dimension of actions, such as Huamoid-v3, TEAC gains significant performance improvement.
We also experimented and compared different settings of T and for their impact on the perfor- mance. As there are no reasonable approaches to set T, we simply compared seven different value levels in the tasks. The results show that the selection of proper 5 value for each task can signifi- cantly boost the performance of TEAC (see more details in Appendix ( and generally r should be smaller for tasks with higher complexity. For example, when we set T 0.001 in Humanoid-v3, TEAC had more than 10% performance gain than that r = 0.005. The reason can be attributed to the stability of the algorithm. For complex tasks, the policy needs more exploration to find the global optima, resulting in larger update steps. Without the help of trust region constraint, the pol- icy will explore arbitrarily in the policy space, losing its bearings and getting trapped in some bad settle-points.
dealing with continuous distributions, the entropy be negative (Abdolmaleki 2016). Hence, 77 should be a small value. We have investigated several heuristic approaches for setting !) provided in MORE, GAC, and SAC, but none of them can serve as a general and effective solution (see more details in Appendix [B. Thus, in our experiments, we simply set 1) as the negative of action space dimension similar to SAC.
For 17,
6 CONCLUSION
In this paper, we propose to integrate two branches of research in RL, trust region methods for better stability and maximum entropy methods for better policy exploration during the learning, benefit from both sides. We first transform the original RI. objective to a constraint optimization problem with the constraints of upper bound KL-divergence to avoid the abrupt difference between the new and old policies and lower bound entropy to promote the policy exploration. Therefore, the Bellman equation redefined accordingly to guide the system loss evaluation. Consequently, we introduce TEAC, an off-colicy algorithm to learn stable and sufficiently explored policies for continuous states and actions. TEAC utilizes two Actor networks to achieve the policy improvement by leveraging the information from the old policy and the exponential of current Q function represented in the critic network. The results show that TEAC with optimized parameters achieves similar performance in half of the tasks and notably improvement in the others terms of efficiency and effectiveness.
REFERENCES
Abbas Abdolmaleki, Rudolf Lioutikov, Nuno Lau, Luís Paulo Reis, Jan Peters, and Gerhard Neu- mann. Model-based relative entropy stochastic search. In Genetic and Evolutionary Computation Conference, GECCO 2016, pp. 153-154, 2016.
Alekh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mahajan. Optimality and approximation with policy gradient methods in markov decision processes. In Jacob Abernethy and Shivani Agarwal (eds.), Proceedings of Thirty Third Conference on Learning Theory, volume 125 of Proceedings of Machine Learning Research, pp. 64-66. PMLR, 09-12 Jul 2020. URL http:
//proceedings mlr. wwwpspeeepppe.e.. wwtp://///aaaaaa/.a html
Riad Akrour, Gerhard Neumann, Hany Abdulsamad, and Abbas Abdolmaleki. Model-free trajectory optimization for reinforcement learning. In Maria-Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, volume 48, pp. 2961-2970, 2016.
Riad Akrour, Abbas Abdolmaleki, Hany Abdulsamad, Jan Peters, and Gerhard Neumann. Model- free trajectory- -bsse policy optimization with monotonic improvement. J. Mach. Learn. Res., 19: 14:1-14:25, 2018
Kamil Ciosek, Quan Vuong, Robert Loftin, and Katja Hofmann. Better exploration with optimistic actor critic. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc,

---

Under review as a conference paper at ICLR 2021
Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, pp. 1785- 1796, 2019.
Christian Daniel, Gerhard Neumann, Oliver Kroemer, and Jan Peters. Hierarchical relative entropy policy search. J. Mach. Learn. Res., 1:99:1-93:50,, 2016.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, ICML 2018, volume 80, pp. 1582-1591. PMLR, 2018.
Tuomas Haarojja Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In Doina Precup and Yee Whye Teh (eds.). Proceedings of the 34th International Conference on Machine Learning, ICML Sydney, NSW, Australia, 6-11 August 2017, volume 70, pp. 1352-1361, 2017.
Tuomas Haarnoja, Vitchyr Pong, Aurick Zhou, Murtaza Dalal, Pieter Abbeel, and Sergey Levine. Composable deep reinforcement learning for robotic manipulation. In 2018 IEEE International Conference on Robotics and Automation, ICRA 2018, Brisbane, Australia, May 21-25, 2018, pp. 6244 6251. IEEE, 201 20.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, ICML 2018, volume 80, pp. 1856-1865, 2018b.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Soft actor-critic algo- rithms and applications. CoRR, abs/1812.05900 2018c.
Sham M Kakade. A natural policy gradient. In Advances in neural information processing systems, pp. 1531-1538, 2002.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, 2015.
Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review. CoRR, Abs/1885.000900 2018.
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuo- motor policies. J. Mach. Learn. Res., 17:39:1-39:40, 2016.
Timothy Lillicrap. Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In Yoshua Bengio and Yann LeCun (eds.), 4th International Conference on Learning Representations, ICLR 2016, 2016.
Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing atari with deep reinforcement learning. CoRR, abs/1312.5602, 2013.
Ofir Nachum, Mohammad Norouzi, Kelvin Xu, and Dale Schuurmans. Bridging the gap between value and policy based reinforcement learning. Advances Neural Information Processing Systems 30: Annual Conference Neural Information Processing Systems 2017, pp. 2775-2785, 2017.
Ofir Nachum, Mohammad Norouzi, Kelvin Xu, and Dale Schuurmans. Trust-pcl: An off-policy trust region method for continuous control. In 6th International Conference Learning Repre- sentations, ICLR 2018. OpenReview.net, 2018.
Jan Peters and Stefan Schaal. Natural actor-critic. Neurocomputing, 71(7-9):1180-1190 -19- 2008. doi: 10. w/w.///.eeccomm..mmmmm 2007. 11.026.
10

---

Under review as a conference paper at ICLR 2021
Jan Peters, Katharina Mülling, and Yasemin Altun. Relative entropy policy search. In Proceedings of the Twenty-Fourth AAAI Conference on Artificial Intelligence. AAAI 2010, 2010.
John Schulman, Sergey Levine, Pieter Abbeel, Michael I. Jordan, and Philipp Moritz. Trust region
policy optimization. In Francis R. Bach and David M. Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, volume 37, pp. 1889-1897, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, hts//707.00347,, 2017.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George van den Driess- che, Julian Schrittwieser, loannis Antonoglou, Vedavyas Panneershelvam,, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of go with deep neural networks and tree search. Nat., 529(7587)44844489, 2016.
Richard Sutton, David Mcallester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. Adv. Neural Inf. Process. Syst, 12, 02 2000.
Richard S. Sutton and Andrew G. Barto. Reinforcement learning an introduction. Adaptive com- putation and machine learning. MIT Press, 1998.
Voot Tangkaratt, Abbas Abdolmaleki, and Masashi Sugiyama. Guide actor-critic for continuous control. In 6th International Conference on Learning Representations, ICLR 2018, 2018
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physies engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2012, pp. 5026-5033. IEEE, 2012.
Mare Toussaint. Robot trajectory optimization using approximate inference. In Andrea Pohoreckyj Danyluk, Léon Bottou, and Michael L. Littman (eds.), Proceedings of the 26th Annual Interna- tional Conference on Machine Learning, ICML 2009, volume 382 of ACM International Confer-
ence Proceeding Series, pp. 1049-1056. ACM, 2009.
Hado van Hasselt, Yotam Doron, Florian Strub, Matteo Hessel, Nicolas Sonnerat, and Joseph Mo- dayil. Deep reinforcement learning and the deadly triad. CoRR, abs/1 18 1e//1812.0264,, 2018.
Brian D. Ziebart, Andrew L. Maas, J. Andrew Bagnell, and Anind K. Dey. Maximum entropy inverse reinforcement learning. In Dieter Fox and Carla Gomes (eds.), Proceedings of the Twenty-Thire AAAI Conference Artificial Intelligence, AAAI 2008, Chicago, Illinois, USA, July 13-17, 2008, pp. 1433 1438.143...11 AAAI Press, 2008

---

Under review as a conference paper at ICLR 2021
APPENDIX
A DERIVATIONS AND PROOFS
A.I TRUST ENTROPY POLICY EVALUATION
Lemma A.1 (Trust Entropy Policy Evaluation). Let = verge to the trust entropy Q-value of n as - oo when considering a mapping Q" S A R with |A] < so and the Bellman backup operator T
T=Q", the sequence Q" will con-
Proof. We define the reward function in trust entropy framework
Tr (s, a.) A r(s, a) (a + 3) Eg'op [aa [x[(|s')]] EEg'o [Ea' (-)a(-
(20)
We rewrite the update rule as
((s - (8, a) + +'laar-pp''' [Q(s', a')l
(21)
Following Sutton & Barto (1988) we can realize the standard convergence for policy evaluation.
A.2 DERIVATION OF THE SOLUTION OF LAGRANGIAN
By taking derivative of C w.r.t. π and setting the derivative to zero,
|/
0,C -Bpls)
(a + () log ((as) + a log -
Ep(o' (M) Ep(a)kaa(((((ss) da - +
(22)
-((s,a) (a B) log (als) + a mal(s)) (V()) (e
- (a + +B++
-0
Given the fact that sample actions from the actor network as parameterized Gaussian distribution and the value function should satisfy Bellman equation, if we set = 0, the function can be rewritten
0 Q= a) (a 3) log π (als) + a log Jou (als) (a A) The solution of π maass) is:
(23)
TT (a 8) (als) añv exp ((8 a)
a 13+3+
exp
(24)
a
Here, we combine the third constraint in Eq. with the solution Eq
24
==Ep(s) f r(aas) s)da
(25)
-Ep(s) f |Tou (a s) पंं
((ssa)) a + 8
a B++ a 3
exp
exp
da
Given the fact that a, 8, and À are constants which independent of and a, can get
/ Told (a 18) aft exp
((s,a) a) a 8
( + A
1 ==Ep(s)
exp
da
a+ 3
(26)
-Ep(s) / Told (a s)at f exp
Q(s,a)
a+
da
exp
a a B
a+B
12

---

Under review as a conference paper at ICLR 2021
Thus,
a a+ -
Bp(s) / Tola (a 8) 2) exp
((s a)
exp
da
(27)
a+
Hence, the term exp
a+++A a+ル
acts normalization term. Therefore,
Q( a) a + B
π <(ass) (ala(a))) att exp
(28)
A.3 TRUST ENTROPY POLICY IMPROVEMENT
Lemma A.2 (Trust Entropy Policy Improvement). Given policy and old policy ã, define
new policy
i(ass) x #(ass) oF3 exp (s,
vs.
(29)
If Q is bounded, J [ exp
( da is bounded for any (for all & and R),
at$
and the policies the parameterized Gaussian networks. Then we obtain Q² (s, 2
Q (s a), Vs, a.
Proof. With the definition of the V function, we can get:
VT
E E 80=
Σ (r((S,;a)) - (a B3 (-gg((1)))) log (t1ee)
(30)
Here, T = (so, on 81, 01,..... denotes the trajectory originating at (s, Using a telescoping argument, we have:
V² (8) (8) -E -B
y - 3) log (-11s) + a log (1s))
(s)
R 8,000
(a B) 7 -(a++) log नं (8x) (-11a))
+ + - V (st)) (+)
@0)
3 8
7 - (a B) log (: (8c) + a log . (-118))
+ +V (51+1) (*((s)))
? ar(s, (a 3) log नं (-11x) + a log (-1ss)
R. 80
+
- v (s:))]
Er 8,00
(10 ax) - (*.) (a+ 3) log म (-1st) + a log π (-1s:))
0
1 1- Ev Ba (Q" V* (a 5 log (-1s' + a log π (-(s'))
(31)
where (a) rearranges terms in the summation and cancels the V(so) term with the V(s) outside
the summation, and (b) uses the tower property of conditional expectations and the final equality follows from the definition of pa (s). Consider Eq 24 can be rewritten as:
T(als) = exp a
log f((ss) + (=
1 (a+++t))
exp
(32)
a
a a
13

---

Under review as a conference paper at ICLR 2021
is normalization term. Assume we follow the gradient ascent update rule and that the distribution p(s) is strictly positive i.c. p(s) > for all states s. Following the work Agarwal et al. 22200) with the help of the gradient of the softmax policy class, we can get
where exp
Σ
{(q)s) - V(s) - (a + B) log (-|ss) + a log x x-|s)) 2 0,
(33)
REA
Then get ($()) (s), as well as Q" (= holds for all states and actions
A.4 DERIVATION OF THE FUNCTION
To obtain the dual function, we take the solution of (als) into Eq.
a,, =B =Bplerx(al (Q(s, a)] a (T Ppp)) KT.
+ + (B 7) + Ep(s) / malls)da-
-B (Q(8, a)]
((s,a)
B+
- (a B(o)
+ a
log (als)
(34)
a+ B
a a+
+ aBBp)))((aa) [log (al((a)))) + Ep(s)) T(als)da-1) 1) + aT - 3mj
3
=07 3m - (ax + 3) Ep(())
a + 3
This loss function be rewritten
a a a+
((a,3) =0t BB + (a + Bp)))))
log
exp
a+
((8 a)
(35)
=07 - Bn + (a + Bp())))
log
(ala(a) (aaaa)) exp
da
+ 3
=y(a, B)
Meanwhile, we can rewrite Eq 24 as
a + 3 138++2 a + B
++ ++
Told (Q]8)
(00 ais exp
Q(a.a)
a+
cxp
(36)
T(als)
With Eq 36 the loss function becomes
Totu(als) ( exp
((x.a) a+
C(a, 3) =at - 3m + (a + 3)EE
log
T(ass)
log Total Totaaaaas (a|s) + Q(8 a)
(37)
=aT - 33 + (a + {))pp())
+ log (als))
a + 3
=07 - -Bn + Ep(e) Ja. log Tala(a)s) + Q(8 a) (a 8) log logloo {(a|s))
14

---

Under review as a conference paper at ICLR 2021
B HYPERPARAMETER ANALYSIS FOR AND 7)
8003
2500
8332
5,000
6202
2000
2002
3392
2.5
2.0
1.5
2-3
2.5
3.5
2.5
2.0
25
326
(a) HalfCheetahh-33
(b) Hopper-v3
(c) Humanoid-v3
140
5800
6000
4000
3000
20
-203
0.5
3.0
3.5
2.0
25
3.3
2.5
Umoologa
(d) Walker2d-v3
(e) Swimmer-v3
(1) Ant-v3
Figure 2: Performance with different 7 on six MuJoCo tasks. T 0.001 achieves about 8000 as the return in Humanoid-v3, and r = 0.005 achieves about 130 as the return in Swimmer-v3. These two results surpass all benchmark algorithms significantly.
The impact of on the performance of TEAC was evaluated with T E (0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1). Fig shows that tuning for different tasks may achieve significant performance improvement in TEAC.
Similar to SAC (Haarnoja al., 2018c), set 7) the negative of action space dimension in own experiments. Besides, there are other techniques in existing methods. MORE (Abdolmalcki al. 2016) changes the entropy constraint to
E- Eo 2 Y (Eold Eo) 17 = (Eold 15) + Eo
(38)
where E x Ed(s) [H (m)(a)))) denotes the expected entropy of the current policy. EoM R3 Ep(a) [H [))] denotes the expected entropy of the old policy, and Eg denotes the entropy of base policy (a 0.011). GAC improves this technique to adjust v heuristically by
1 = max (7 (E - Eo) + Eo, Eo).
(39)
We compared these three different techniques Hopper-v3, Humanoid-v3 and Ant-v3. Fig, shows that there is no outstanding difference from them. Therefore, we simply set r7 as the negative of action space dimension similar SAC.
3530
0000
1930
4000
4232
8
2003
2000
1890
330
1806
2000 2013
(a) H3pper-v3
(b) Humanoid-v3
(c) Ant-v3
Figure 3: Performance with different in three Mujoco tasks with 7 = 0.1.
15

---

Under review as a conference paper at ICLR 2021
C
ALGORITHMS
Algorithm TEAC: Trust Entropy Actor Critic
Input:
Initial actor To s). old actor Ta(a(s),
critic Qo and Q+ target critic network Φ1 +- Φ1, Φ2 t 02
KL divergence bound т, entropy bound 17, learning rate waos Was WB.
empty replay pool D Ø
1: for each iteration do
2: for each environment step do
3: Observe state 8, and sample action at (T((a(()))
4: Execute (e, receive reward r(8:, a.) and next state 8/ ~ p(1//11. a:)
5: Add transition to replay buffer D {(St, a (8c, ar) s.]]
6: end for
7: for each gradient step do
8: Sample N mini-batch samples {(s; } uniformly from D
9:
Sample actions a/ ~ TT(aass), compute Qua (5 a(): Quris', a!) min(Qs, a)) Qs (8/, a!))
(40)
10:
Compute you update @ by, e.g., Adam, and update by moving average:
1/1 Ti + 7 (Quar (8', a() (a + log ( + logo logmm())ss)
(41)
N
oj - 6j Wac
( oy (8,, a.) -1)² .j E (1.2)
(42)
क, KØj +
x) E [1.2]
(43)
11:
Sample actions à To (als;) compute Q( a):
Q(s..a) = min (Q,, (s; a), Qai (8:, ā))
(44)
Compute loss function of A:
((0) Σ
((a + log ((als)) - a log (a(as))) Q(s;,
(45)
13:
Update á using < 0
Update 0 by, e.g., Adam:
0 - Who VoL(O)
(46)
15:
Compute loss function of dual variables a and B:
N
((a) =
a (loge((a)s)) - log (a)ss)) -)
(47)
N
1
C(B)
3 (log To(als;) + n)
(48)
N
16: Update dual variables a and 8:
a Wa V.L(a), - ((((()
(49)
17: end for
18: end for
16

---

Under review as a conference paper at ICLR 2021
D HYPERPARAMETERS
Table lists the effective hyperparameters of TEAC used in the experiments, of which the results shown Fig.
Table 1: TEAC Hyperparameters
Parameter
Value
optimizer
Adam
Kingma & Ba 2015)
learning rate for actor and critic
10
learning rate for
1 1-10-
learning rate for B
1- 1.10-3
discount (?)
0.99
replay buffer size number of hidden layers (all networks) number of hidden units per layer number samples per minibatch
10
256
100
target entropy (1)
dim(AA))e.g. -6 for HalfCheetah-v3)
max divergence for KL (T)
0.005
nonlinearity
ReLU
target smoothing coefficient K target update interval gradient steps
0.005
E THE BEST PERFORMANCE OF OUR MODEL
1000
2500
66032
4003
15:00
4392
3000
2303
2.0
2.5
3.5 Simanass
simestopa
(a) HalfCheetah-v33
(b) Hopper-v3
(c) Humanoid-v3
60:0
$0.0
120
5000
3000
2000
2000
2000
20
-2000
106
3P4
106
(d) W3lker2d-v3
(c) Swimmer-v3
(f) Ant-v3
Figure 4: Performance comparisons on six MuJoCo tasks. Notice that the blue line the perfor- mance of our model which setting different with respect different tasks. In this figure, we set - = 0.5 for HalfChectah-v3, J 0.05 for Ant-v3, т = 0.1 for Hopper-v3, T 0.001 for Humanoid-v3, T 0.005 for Swimmer-v3, and 0.1 for Walker2d-v3.
*Our implementation also makes use of two Q-functions (critic networks) to mitigate positive bias in the policy improvement step, follows (allloosFFjjmttooccooooo et al. (2018) and Haarnoja al. (2018)).
17

---

Under review as a conference paper at ICLR 2021
F ADDITIONAL CONTINUOUS CONTROL EXPERIMENTS WITH SIX INSTANCES
04003
7903
3000
10803
2000
4338
4332
3333
2332
2.5
2.33
1.0
305
106
HalfCheetah-v3
(h) Hopper-v3
(c) Humanoid-v3
140
50.00
2000
4000
1n0
5800
3000
-2000
7.9
2.0
05
3.0
10
X.0
2/1
(d) Walker2d-v3
(c) Swimmer-v3
(f) Ant-v3
Figure 5: Performance comparisons on six MuJoCo tasks. We trained six different instances of all algorithms with different random seeds. In this case, for TEAC, we set as the negative of action space dimension of the task, and set 0.005 for all tasks.
18


Reviews:
[{"review_id": "zdjzn4DTdu", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper addresses the problem of reinforcement learning in continuous spaces by formulating the problem as a constrained optimization problem. In this problem, the objective is maximizing the expected reward, and the constraints ensure that 1) the distance between the new and old policies is bounded, 2) the entropy is above a threshold, and 3) the assumptions of MDP hold. The paper then derives closed-form solutions for the Lagrangian, which are used for obtaining variable update rules. An empirical study examines the ideas against benchmark problems. \n\nThe paper has a good structure and style. The claims seem sound and the results look promising. However, since I'm not familiar with the topic, I cannot verify the details of those claims and results. I base my recommendation on a high-level idea of the paper. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "xN-q_LKU8mh", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose to add three more constraints to the well-known trust region policy optimization model. The first one of these constraints aims at keeping the entropy level higher than a given threshold. The authors then relax these constraints to obtain the Lagrangian function. An approach using primal-dual updates is proposed to obtain a solution.\n\nIn general, I find the paper difficult to follow and hence cannot properly assess its novelty. My comments and questions are as follows:\n\n- The function Q^\\pi in (1) is not defined at that stage.\n\n- How can we guarantee the feasibility of model (2) for different values of \\eta? Likewise, if we obtain a solution with tackling the Lagrangian function, then is that solution feasible for (2)?\n\n- How do you go from the first line to the second line in relation (4)?\n\n- Why \\lambda is in (5)?\n\n- What is Q-\\phi in (8)?\n\n- What does setting \\eta to dimension in Table 1 signify? ", "extra_scores": {}, "extra_text": {}}, {"review_id": "nVfPqhpmdx_", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Previous work such as MOTO and GAC has developed approaches that combine trust regions for policy stability with maximum entropy to ensure adequate exploration.  This paper introduces an algorithm called TEAC in this spirit which adds several technical novelties to address limitations of previous work:\n - It uses a particular Gaussian policy parameterization for the actor to ensure that updated policies are easily representable in the same class while prior approaches used more complex solutions.\n - It uses a modified objective for the critic that incorporates the trust region and entropy terms\n - It uses a first (rather than second) order method to optimize the dual variables\n - It uses these to provide a proof of policy improvement.\nTEAC is evaluated on a number of MuJoCo tasks.\n\nI think the TEAC algorithm and theoretical analysis are a nice contribution and that ultimately this will be a nice paper.  However, I have some concerns with the positioning of the contribution, the rigor of the technical exposition, and the experiments.\n\n1) The introduction makes it seem like the contribution of the paper is proposing the idea of combining trust regions and maximum entropy, but as is discussed later this idea is already present in previous work (e.g MOTO and GAC).  The real contribution is discussed only in very compressed form in the last paragraph.  Having a fuller discussion in Section 4 after the technical exposition is reasonable, but the key ideas and problems the paper solves to make this approach work better than previous work should at least by clearly stated and some intuition or motivation provided in the introduction.\n\n2) In Equation (2), something about the notation for the =1 constraint doesn’t seem quite right.  The expectation isn’t being taken over anything.  This issue persists into 3.  The notation for the last MDP constraint seems a bit odd as well, although I supposes the intent is to have s’ drawn from the resulting distribution.\n\n3) Equation 5 still has \\lambda in it despite setting it to 0.\n\n4) Lemma 2: the old policy \\hat{\\pi} is being given, not defined right?  At least I don’t see a definition of it.\n\n5) The proof of Lemma 2 should point to A.3 not A.2.  I’m not sure why A.2 exists.  It doesn’t seem to be otherwise referenced and seems to be a repetition of material in the main text.\n\n6) The reason (27) is true should be explained rather than point generally to another paper without an specific explanation.  I also don’t understand why (28) is an application of (6) and there is an ellipsis here.  I assume what is going on is an argument about what happens as the Bellman operator is repeatedly applied as it approaches its limit, which we know from (10) is the final quantity.\n\n7) Given the close relationship, it seems odd that no comparison to MOTO is made.  GAC does not appear to provide one either, so even an indirect comparison does not appear to exist.  Furthermore GAC mysteriously disappears after Figure 1(b), which is problematic as it appears to be the most closely related work in terms of technique.\n\n8) The plots only show three trials of each algorithm, which limits the confidence of performance assessments based on them and is not what I would describe as “extensive” in the abstract and elsewhere.  While I agree based on the results shown that TEAC typically outperforms the non-SAC algorithms, the comparison with SAC is more mixed except for the strong performance in swimmer (although even here the strong performance of PPO seems similar to TEAC).  So I would tone down the unqualified claim that “TEAC outperforms the state-of-the-art solutions”.\n\nUpdates after author responses and discussion:\n(1) After the updates, most of the positioning issues have been improved\n(2) The technical exposition is improved, and the main argument I was bothered by is somewhat improved, though it still has a big jump near the end.\n(3) I'm still bothered that to demonstrate improvement the algorithm is tuned on a per-example basis but the baselines are not.  The results certainly show that the former is reasonable, but to make the comparison fair the latter needs to be done as well.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "HHNr4zcdQvc", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes Trust Entropy Actor Critic (TEAC), a novel algorithm for reinforcement learning (RL) combining the idea of TRPO/PPO and max-entropy RL, together with the corresponding critic, actor and dual updates. The high level idea is that trust region methods ensure stability by constraining the KL divergence from the previous policy, while entropy regularization encourages exploration, and hence combining the two may achieve the best of both worlds and obtain a good trade-off between stability and exploration. To achieve this goal, the authors propose to augment the original trust-region subproblem in TRPO with an additional constraint on the lower bound of the policy entropy (together with two other trivial constraints corresponding to the validity of the policy in the MDP framework). Then by forming the Lagrangian function and setting the gradient to zero, the authors obtain both the critic (value) and actor (policy) updates with different choices of dual variables (corresponding to the two trivial constraints), together with the dual updates. Numerical experiments compared to some popular baseline RL algorithms are also reported to demonstrate the improvement of TEAC compared to the existing works. \n\nin general, the high level idea is interesting and reasonable, and some empirical improvement is also shown. However, there are several undefined terms and technical issues/mistakes that largely downgrade the quality and contribution of this paper. \n1. $Q^{\\pi}$ in (1) and (2) is not defined, and the meaning of $E_{\\rho_{\\pi}(s),\\pi(a|s)}$ is unclear. Also, according to standard literature like [1, Chapter 4], the expected reward $\\mathcal{J}(\\pi)$ should indeed be something like $\\frac{1}{1-\\gamma}E_{s\\sim \\rho_{\\pi}, a\\sim \\pi(\\cdot|s)}r(s,a)$, so the term inside the expectation should probably be the reward $r(s,a)$ instead of the Q function $Q^{\\pi}(s,a)$.\n2. The formulation in (2) is confusing. Firstly, it is unclear what the notation $E_{\\rho_{\\pi}(s)\\pi(a|s)}=1$ means, and what the difference is between $E_{\\rho_{\\pi}(s)\\pi(a|s)}$ and $E_{\\rho_{\\pi}(s),\\pi(a|s)}$. Similarly, it is unclear what $V$ is (as a function of $\\pi$ or a free variable), and what the notation $E_{\\rho_{\\pi}(s)\\pi(a|s)p(s’|s,a)}$ means. In general, all the terms appearing in the optimization problem should either be some constants or a function of $\\pi$, but this is not made clear by the authors. Also, it’s unclear why the authors switch the orders of $\\pi$ and $\\pi_{\\rm old}$ in the KL divergence compared to the original TRPO formulation, and some explanations should be provided. \n3. The formula (4) seems to be weird. Firstly, according to the right-hand side, it seems that the left-hand side should indeed be the (s,a)-entry of the gradient instead of the entire gradient. Secondly, it is unclear why $\\rho(s)$ disappears in the second equality. \n4. The authors argue that the dual variables $\\nu$ and $\\lambda$ can be taken arbitrarily as the policy is parametrized as a Gaussian via neural networks so that it is always a valid randomized policy (satisfying the third constraint in (2)), while the “value function” always satisfies the Bellman equation corresponding to the fourth constraint in (2). However, with the neural network parametrization $\\theta$, the optimization problem should also be solved w.r.t. $\\theta$ instead of $\\pi$, in which case (4) no longer makes sense (since it’s differentiating w.r.t. $\\pi$ instead of $\\theta$). Also, as mentioned above, the value function $V$ is undefined and hence the second claim does not make much sense. In addition, if the fourth constraint in (2) corresponds to the Bellman equation, then it is weird why no reward $r$ or discount factor $\\gamma$ is involved there. \n5. In Lemma 1, the “trust entropy Q-value” is undefined. \n6. In (10), why can $\\exp(-(\\alpha+\\beta+\\lambda)/(\\alpha+\\beta))$ serve as a normalization term? In particular, why does it hold that $\\int_a\\pi_{\\rm old}(a|s)^{\\alpha/(\\alpha+\\beta)}\\exp(Q^{\\pi}(s,a)/(\\alpha+\\beta))da$ always equal to $\\exp(-(\\alpha+\\beta+\\lambda)/(\\alpha+\\beta))$ for any state $s$?\n\nAlso, the experimental part has some limitations.\n1. The numerical improvement is not very convincing, as it seems that TEAC only improves over SAC in two out of six tasks and only has significant improvement on one of these two tasks. Some more thorough comparisons are needed to validate the empirical advantage of the proposed method. \n2. The actually implemented algorithm, Algorithm 1 contains several tricks unexplained in the main text. In particular, why do we need two target critic networks? And in the update (38), should $\\bar{\\phi}$ be $\\bar{\\phi}_i$? \n\nFinally, please find some miscellaneous suggestions/comments on additional references, technical issues fixing and typo fixing below.\n1. The work [2] seems to be closely related to the high level idea of this paper, and should better be compared with. \n2. In the abstract, “transforms” should be “transform”. In the first paragraph of the introduction, “learning process” should be “learning processes”. \n3. In the last paragraph of the introduction, “guaranteed to converge” is not very accurate. In fact, only the critic/policy evaluation is guaranteed to converge, and the authors show that policy improvement does hold (but may or may not lead to eventual convergence of the whole TEAC algorithm). \n4. In the first paragraph of Section 2, the definition of $\\rho_{\\pi}$ is not provided, although from the literature it probably means the discounted state-visitation distribution/measure. The authors should provide a clear definition for self-contained-ness, and the term “state of the trajectory distribution” does not seem to make much sense and should better be replaced with more standard terminologies like “discounted state-visitation distribution/measure”. Also, “qantified” should be “quantified”. \n5. In TEAC, the parameters $\\tau$ and $\\eta$ are always fixed. However, as the algorithm proceeds, it may not make much sense to keep a constant exploration power as required by the constant $\\eta>0$. Will a decreasing $\\eta$ be a better choice? Some discussions on this should better be provided. \n6. In (3), $\\rho(s)$ should be $\\rho_{\\pi}(s)$, and the dual variables $\\alpha$ and $\\beta$ should be required to be non-negative. Accordingly, the dual updates should probably better be projected onto the non-negative orthant. \n7. In the sentence before (4), “derivation” should be “derivative”. \n8. In (5), $Q$ should be $Q^{\\pi}$. \n9. In (20), the entropy terms lack right parentheses. \n\nHence in general, I think the paper is still not ready for publication and needs substantial fixing and improvement. \n\n[1] Agarwal, Alekh, Nan Jiang, and Sham M. Kakade. Reinforcement learning: Theory and algorithms. Technical Report, Department of Computer Science, University of Washington, 2019.\n\n[2] Pajarinen, Joni, Hong Linh Thai, Riad Akrour, Jan Peters, and Gerhard Neumann. \"Compatible natural gradient policy search.\" Machine Learning 108, no. 8-9 (2019): 1443-1466.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a reinforcement learning algorithm that combines trust region policy optimization and entropy maximization. The starting point is the Lagrangian of a constrained optimization problem that upper bounds the change in the policy and lower bounds the entropy of the policy. The paper proves that the algorithm converges, and evaluates it experimentally in MuJoCo domains.

The main issues raised by the reviewers were related to the proofs (see especially R1) and experimental evaluation (R4). The authors did a great job improving the paper during the discussion phase, but some of the issues remain unresolved, and thus reviewers find the paper not to be ready for publication. Thus, I'm recommending rejection.
Decision: Reject