Venue: ICLR 2018 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper introduces a modified actor-critic algorithm where a “guide actor” uses approximate second order methods to aid computation. The experimental results are similar to previously proposed methods. 

The paper is fairly well-written, provides proofs of detailed properties of the algorithm, and has decent experimental results. However, the method is not properly motivated. As far as I can tell, the paper never answers the questions: Why do we need a guide actor? What problem does the guide actor solve? 

The paper argues that the guide actor allows to introduce second order methods, but (1) there are other ways of doing so and (2) it’s not clear why we should want to use second-order methods in reinforcement learning in the first place. Using second order methods is not an end in itself. The experimental results show the authors have found a way to use second order methods without making performance *worse*. Given the high variability of deep RL, they have not convincingly shown it performs better.

The paper does not discuss the computational cost of the method. How does it compare to other methods? My worry is that the method is more complicated and slower than existing methods, without significantly improved performance.

I recommend the authors take the time to make a much stronger conceptual and empirical case for their algorithm. 

review_title: Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: 

The authors devise and explore use of the hessian of the
(approximate/learned) value function (the critic) to update the policy
(actor) in the actor-critic  approach to RL.  They connect their
technique, 'guide actor-critic' or GAC, to existing actor-critic
methods (authors claim only two published work use 1st order
information on the critic). They show that the 2nd order information
can be useful (in several of the 9 tasks, their GAC techniques were
best or competitive, and in only one, performed poorly compared to best).

The paper has a technical focus.

pros:

- Strict generalization of an existing (up to 1st order) actor-critic approaches.

- Compared to many existing techniques, on 9 tasks

cons:

- no mention of time costs, except that for more samples, S > 1, for
 taylor approximation, it can be very expensive.

- one would expect more information to strictly improve performance,
  but the results are a bit mixed (perhaps due to convergence to local
  optima and both actor and critic being learned at same time, 
  or the Gaussian assumptions, etc.).

- relevance: the work presents a new approach to actor-critique strategy for
  reinforcement learning, remotely related to 'representation
  learning' (unless value and policies are deemed a form of
  representation).


Other comments/questions:

- Why does the performance start high on Ant (1000), then goes to 0
(all approaches)?

- How were the tasks selected? Are they all the continuous control
  tasks available in open ai?


 





review_title: A promising technique, using hessian of the critic, for learning the actor, for continuous control, with competitive results 

final_score: rating: 7: Good paper, accept
review_content: The paper presents a clever trick for updating the actor in an actor-critic setting: computing a guide actor that diverges from the actor to improve critic value, then updating the actor parameters towards the guide actor. This can be done since, when the parametrized actor is Gaussian and the critic value can be well-approximated as quadratic in the action, the guide actor can be optimized in closed form.

The paper is mostly clear and well-presented, except for two issues: 1) there is virtually nothing novel presented in the first half of the paper (before Section 3.3); and 2) the actual learning step is only presented on page 6, making it hard to understand the motivation behind the guide actor until very late through the paper.

The presented method itself seems to be an important contribution, even if the results are not overwhelmingly positive. It'd be interesting to see a more elaborate analysis of why it works well in some domains but not in others. More trials are also needed to alleviate any suspicion of lucky trials.

There are some other issues with the presentation of the method, but these don't affect the merit of the method:

1. Returns are defined from an initial distribution that is stationary for the policy. While this makes sense in well-mixing domains, the experiment domains are not well-mixing for most policies during training, for example a fallen humanoid will not get up on its own, and must be reset.

2. The definition of beta(a|s) as a mixture of past actors is inconsistent with the sampling method, which seems to be a mixture of past trajectories.

3. In the first paragraph of Section 3.3: "[...] the quality of a guide actor mostly depends on the accuracy of Taylor's approximation." What else does it depend on? Then: "[...] the action a_0 should be in a local vicinity of a."; and "[...] the action a_0 should be similar to actions sampled from pi_theta(a|s)." What do you mean "should"? In order for the Taylor approximation to be good?

4. The line before (19) is confusing, since (19) is exact and not an approximation. For the approximation (20), it isn't clear if this is a good approximation. Why/when is the 2nd term in (19) small?

5. The parametrization nu of \hat{Q} is never specified in Section 3.6. This is important in order to evaluate the complexities involved in computing its Hessian.

review_title: Clever approach

Meta-review: The reviewers agree that the formulation is novel and interesting, but they raised concerns regarding the motivation and the complexity of the approach. I find the authors' response mostly satisfying, and I ask them to improve the paper by incorporating the comments.

Detailed comments:
The maximum-entropy objective used in Eq. (13) reminds me of maximum-entropy RL objective in previous work including [Ziebart, 2010], [Azar, 12], [Nachum, 2017], and [Haarnoja, 2017].
Decision: Accept (Poster)