Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes Trust Entropy Actor Critic (TEAC), a novel algorithm for reinforcement learning (RL) combining the idea of TRPO/PPO and max-entropy RL, together with the corresponding critic, actor and dual updates. The high level idea is that trust region methods ensure stability by constraining the KL divergence from the previous policy, while entropy regularization encourages exploration, and hence combining the two may achieve the best of both worlds and obtain a good trade-off between stability and exploration. To achieve this goal, the authors propose to augment the original trust-region subproblem in TRPO with an additional constraint on the lower bound of the policy entropy (together with two other trivial constraints corresponding to the validity of the policy in the MDP framework). Then by forming the Lagrangian function and setting the gradient to zero, the authors obtain both the critic (value) and actor (policy) updates with different choices of dual variables (corresponding to the two trivial constraints), together with the dual updates. Numerical experiments compared to some popular baseline RL algorithms are also reported to demonstrate the improvement of TEAC compared to the existing works. 

in general, the high level idea is interesting and reasonable, and some empirical improvement is also shown. However, there are several undefined terms and technical issues/mistakes that largely downgrade the quality and contribution of this paper. 
1. $Q^{\pi}$ in (1) and (2) is not defined, and the meaning of $E_{\rho_{\pi}(s),\pi(a|s)}$ is unclear. Also, according to standard literature like [1, Chapter 4], the expected reward $\mathcal{J}(\pi)$ should indeed be something like $\frac{1}{1-\gamma}E_{s\sim \rho_{\pi}, a\sim \pi(\cdot|s)}r(s,a)$, so the term inside the expectation should probably be the reward $r(s,a)$ instead of the Q function $Q^{\pi}(s,a)$.
2. The formulation in (2) is confusing. Firstly, it is unclear what the notation $E_{\rho_{\pi}(s)\pi(a|s)}=1$ means, and what the difference is between $E_{\rho_{\pi}(s)\pi(a|s)}$ and $E_{\rho_{\pi}(s),\pi(a|s)}$. Similarly, it is unclear what $V$ is (as a function of $\pi$ or a free variable), and what the notation $E_{\rho_{\pi}(s)\pi(a|s)p(s’|s,a)}$ means. In general, all the terms appearing in the optimization problem should either be some constants or a function of $\pi$, but this is not made clear by the authors. Also, it’s unclear why the authors switch the orders of $\pi$ and $\pi_{\rm old}$ in the KL divergence compared to the original TRPO formulation, and some explanations should be provided. 
3. The formula (4) seems to be weird. Firstly, according to the right-hand side, it seems that the left-hand side should indeed be the (s,a)-entry of the gradient instead of the entire gradient. Secondly, it is unclear why $\rho(s)$ disappears in the second equality. 
4. The authors argue that the dual variables $\nu$ and $\lambda$ can be taken arbitrarily as the policy is parametrized as a Gaussian via neural networks so that it is always a valid randomized policy (satisfying the third constraint in (2)), while the “value function” always satisfies the Bellman equation corresponding to the fourth constraint in (2). However, with the neural network parametrization $\theta$, the optimization problem should also be solved w.r.t. $\theta$ instead of $\pi$, in which case (4) no longer makes sense (since it’s differentiating w.r.t. $\pi$ instead of $\theta$). Also, as mentioned above, the value function $V$ is undefined and hence the second claim does not make much sense. In addition, if the fourth constraint in (2) corresponds to the Bellman equation, then it is weird why no reward $r$ or discount factor $\gamma$ is involved there. 
5. In Lemma 1, the “trust entropy Q-value” is undefined. 
6. In (10), why can $\exp(-(\alpha+\beta+\lambda)/(\alpha+\beta))$ serve as a normalization term? In particular, why does it hold that $\int_a\pi_{\rm old}(a|s)^{\alpha/(\alpha+\beta)}\exp(Q^{\pi}(s,a)/(\alpha+\beta))da$ always equal to $\exp(-(\alpha+\beta+\lambda)/(\alpha+\beta))$ for any state $s$?

Also, the experimental part has some limitations.
1. The numerical improvement is not very convincing, as it seems that TEAC only improves over SAC in two out of six tasks and only has significant improvement on one of these two tasks. Some more thorough comparisons are needed to validate the empirical advantage of the proposed method. 
2. The actually implemented algorithm, Algorithm 1 contains several tricks unexplained in the main text. In particular, why do we need two target critic networks? And in the update (38), should $\bar{\phi}$ be $\bar{\phi}_i$? 

Finally, please find some miscellaneous suggestions/comments on additional references, technical issues fixing and typo fixing below.
1. The work [2] seems to be closely related to the high level idea of this paper, and should better be compared with. 
2. In the abstract, “transforms” should be “transform”. In the first paragraph of the introduction, “learning process” should be “learning processes”. 
3. In the last paragraph of the introduction, “guaranteed to converge” is not very accurate. In fact, only the critic/policy evaluation is guaranteed to converge, and the authors show that policy improvement does hold (but may or may not lead to eventual convergence of the whole TEAC algorithm). 
4. In the first paragraph of Section 2, the definition of $\rho_{\pi}$ is not provided, although from the literature it probably means the discounted state-visitation distribution/measure. The authors should provide a clear definition for self-contained-ness, and the term “state of the trajectory distribution” does not seem to make much sense and should better be replaced with more standard terminologies like “discounted state-visitation distribution/measure”. Also, “qantified” should be “quantified”. 
5. In TEAC, the parameters $\tau$ and $\eta$ are always fixed. However, as the algorithm proceeds, it may not make much sense to keep a constant exploration power as required by the constant $\eta>0$. Will a decreasing $\eta$ be a better choice? Some discussions on this should better be provided. 
6. In (3), $\rho(s)$ should be $\rho_{\pi}(s)$, and the dual variables $\alpha$ and $\beta$ should be required to be non-negative. Accordingly, the dual updates should probably better be projected onto the non-negative orthant. 
7. In the sentence before (4), “derivation” should be “derivative”. 
8. In (5), $Q$ should be $Q^{\pi}$. 
9. In (20), the entropy terms lack right parentheses. 

Hence in general, I think the paper is still not ready for publication and needs substantial fixing and improvement. 

[1] Agarwal, Alekh, Nan Jiang, and Sham M. Kakade. Reinforcement learning: Theory and algorithms. Technical Report, Department of Computer Science, University of Washington, 2019.

[2] Pajarinen, Joni, Hong Linh Thai, Riad Akrour, Jan Peters, and Gerhard Neumann. "Compatible natural gradient policy search." Machine Learning 108, no. 8-9 (2019): 1443-1466.
review_title: An interesting idea with several technical issues

final_score: rating: 5: Marginally below acceptance threshold
review_content: Previous work such as MOTO and GAC has developed approaches that combine trust regions for policy stability with maximum entropy to ensure adequate exploration.  This paper introduces an algorithm called TEAC in this spirit which adds several technical novelties to address limitations of previous work:
 - It uses a particular Gaussian policy parameterization for the actor to ensure that updated policies are easily representable in the same class while prior approaches used more complex solutions.
 - It uses a modified objective for the critic that incorporates the trust region and entropy terms
 - It uses a first (rather than second) order method to optimize the dual variables
 - It uses these to provide a proof of policy improvement.
TEAC is evaluated on a number of MuJoCo tasks.

I think the TEAC algorithm and theoretical analysis are a nice contribution and that ultimately this will be a nice paper.  However, I have some concerns with the positioning of the contribution, the rigor of the technical exposition, and the experiments.

1) The introduction makes it seem like the contribution of the paper is proposing the idea of combining trust regions and maximum entropy, but as is discussed later this idea is already present in previous work (e.g MOTO and GAC).  The real contribution is discussed only in very compressed form in the last paragraph.  Having a fuller discussion in Section 4 after the technical exposition is reasonable, but the key ideas and problems the paper solves to make this approach work better than previous work should at least by clearly stated and some intuition or motivation provided in the introduction.

2) In Equation (2), something about the notation for the =1 constraint doesn’t seem quite right.  The expectation isn’t being taken over anything.  This issue persists into 3.  The notation for the last MDP constraint seems a bit odd as well, although I supposes the intent is to have s’ drawn from the resulting distribution.

3) Equation 5 still has \lambda in it despite setting it to 0.

4) Lemma 2: the old policy \hat{\pi} is being given, not defined right?  At least I don’t see a definition of it.

5) The proof of Lemma 2 should point to A.3 not A.2.  I’m not sure why A.2 exists.  It doesn’t seem to be otherwise referenced and seems to be a repetition of material in the main text.

6) The reason (27) is true should be explained rather than point generally to another paper without an specific explanation.  I also don’t understand why (28) is an application of (6) and there is an ellipsis here.  I assume what is going on is an argument about what happens as the Bellman operator is repeatedly applied as it approaches its limit, which we know from (10) is the final quantity.

7) Given the close relationship, it seems odd that no comparison to MOTO is made.  GAC does not appear to provide one either, so even an indirect comparison does not appear to exist.  Furthermore GAC mysteriously disappears after Figure 1(b), which is problematic as it appears to be the most closely related work in terms of technique.

8) The plots only show three trials of each algorithm, which limits the confidence of performance assessments based on them and is not what I would describe as “extensive” in the abstract and elsewhere.  While I agree based on the results shown that TEAC typically outperforms the non-SAC algorithms, the comparison with SAC is more mixed except for the strong performance in swimmer (although even here the strong performance of PPO seems similar to TEAC).  So I would tone down the unqualified claim that “TEAC outperforms the state-of-the-art solutions”.

Updates after author responses and discussion:
(1) After the updates, most of the positioning issues have been improved
(2) The technical exposition is improved, and the main argument I was bothered by is somewhat improved, though it still has a big jump near the end.
(3) I'm still bothered that to demonstrate improvement the algorithm is tuned on a per-example basis but the baselines are not.  The results certainly show that the former is reasonable, but to make the comparison fair the latter needs to be done as well.

review_title: An interesting algorithm and and analysis, but aspects of the paper need improvement

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors propose to add three more constraints to the well-known trust region policy optimization model. The first one of these constraints aims at keeping the entropy level higher than a given threshold. The authors then relax these constraints to obtain the Lagrangian function. An approach using primal-dual updates is proposed to obtain a solution.

In general, I find the paper difficult to follow and hence cannot properly assess its novelty. My comments and questions are as follows:

- The function Q^\pi in (1) is not defined at that stage.

- How can we guarantee the feasibility of model (2) for different values of \eta? Likewise, if we obtain a solution with tackling the Lagrangian function, then is that solution feasible for (2)?

- How do you go from the first line to the second line in relation (4)?

- Why \lambda is in (5)?

- What is Q-\phi in (8)?

- What does setting \eta to dimension in Table 1 signify? 
review_title: Hard to follow work proposing a trust region policy optimization model with additional constraints

final_score: rating: 7: Good paper, accept
review_content: The paper addresses the problem of reinforcement learning in continuous spaces by formulating the problem as a constrained optimization problem. In this problem, the objective is maximizing the expected reward, and the constraints ensure that 1) the distance between the new and old policies is bounded, 2) the entropy is above a threshold, and 3) the assumptions of MDP hold. The paper then derives closed-form solutions for the Lagrangian, which are used for obtaining variable update rules. An empirical study examines the ideas against benchmark problems. 

The paper has a good structure and style. The claims seem sound and the results look promising. However, since I'm not familiar with the topic, I cannot verify the details of those claims and results. I base my recommendation on a high-level idea of the paper. 
review_title: Possibly a good paper

Meta-review: The paper proposes a reinforcement learning algorithm that combines trust region policy optimization and entropy maximization. The starting point is the Lagrangian of a constrained optimization problem that upper bounds the change in the policy and lower bounds the entropy of the policy. The paper proves that the algorithm converges, and evaluates it experimentally in MuJoCo domains.

The main issues raised by the reviewers were related to the proofs (see especially R1) and experimental evaluation (R4). The authors did a great job improving the paper during the discussion phase, but some of the issues remain unresolved, and thus reviewers find the paper not to be ready for publication. Thus, I'm recommending rejection.
Decision: Reject