Venue: ICLR 2017 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper looks at several innovations for deep RL, and evaluates their effect on solving games in the Atari domain.  The paper reads a bit like a laundry list of the researcher’s latest tricks.  It is written clearly enough, but lacks a compelling message.  I expect the work will be interesting to people already implementing deep RL methods, but will probably not get much attention from the broader community.

The claims on p.1 suggest the approach is stable and sample efficience, and so I expected to see some theoretical analysis with respect to these properties. But this is an empirical claim; it would help to clarify that in the abstract.

The proposed innovations are based on sound methods.  It is particularly nice to see the same approach working for both discrete and continuous domains.

The paper has reasonably complete empirical results. It would be nice to see confidence intervals on more of the plots. Also, the results don’t really tease apart the effect of each of the various innovations, so it’s harder to understand the impact of each piece and to really get intuition, for example about why ACER outperforms A3C.  Also, it wasn’t clear to me why you only get matching results on discrete tasks, but get state-of-the-art on continuous tasks.

The paper has good coverage of the related literature. It is nice to see this work draw more attention to Retrace, including the theoretical characterization in Sec.7.

review_title: Review

final_score: rating: 7: Good paper, accept
review_content: This paper studies the off-policy learning of actor-critic with experience replay. This is an important and challenging problem in order to improve the sample efficiency of the reinforcement learning algorithms. The paper attacks the problem by introducing a new way to truncate importance weight, a modified trust region optimization, and by combining retrace method. The combination of the above techniques performs well on Atari and MuJoCo in terms of improving sample efficiency. My main comment is how does each of the technique contribute to the performance gain? If some experiments could be carried out to evaluate the separate gains from these tricks, it would be helpful. 
review_title: null

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper introduces an actor-critic deep RL approach with experience replay, which combines truncated importance sampling and trust region policy optimization. The paper also proposes a new method called stochastic duelling networks to estimate the critic for continuous action spaces. The method is applied to Atari games and continuous control problems, where it yields performance comparable to state-of-the-art methods.

As mentioned in the beginning of the paper, the main contributions of this work lies in combining 1) truncated importance sampling with retrace, 2) trust region policy optimization, and 3) stochastic duelling networks. These improvements work well and may be beneficial to future work in RL.

However, each improvement appears to be quite incremental. Moreover, the ACER framework seems much more complex and fragile to implement compared to the standard deep q-learning with prioritized replay (which appears to perform just as well on Atari games). So for the Atari domain, I would still put my money on prioritized replay due to its simplicity. Thirdly, improving sample efficiency for deep RL is a laudable goal, but really this goal should be pursued in a problem setting where sample efficiency is important. Unfortunately, the paper only evaluates sample efficiency in the Atari and continuous control tasks domain; two domains where sample efficiency is not important. Thus, it is not clear that the proposed method ACER will generalize to problems where we really care about sample efficiency.

Some technical aspects which need clarifications:
- For Retrace, I assume that you compute recursively $Q^{ret}$ starting from the end of each trajectory? Please comment on this.
- It's not clear to me how to derive eq. (7). Is an approximation (double tilde) sign missing?
- In section 3.1 the paper argued that $Q^{ret}$ gives a lower-variance estimate of the action-value function. Then why not use it in eq. (8) for the bias correction term?
- The paper states that it uses a replay memory of 50000 frames, so that across threads it is comparable in size to previous work. However, for each thread this is much smaller compared to earlier experiments on Atari games. For example, one million experience replay transitions were used in the paper "Prioritized Experience Replay" by Schaul et al. This may have a huge impact on performance of the models (both for ACER and for the competing models). In order to properly assess the improvements of ACER over previous work, the authors need to also experiment with larger experience replay memories.


Other comments:
- Please move Section 7 to the appendix.
- "Moreover, when using small values of lambda to reduce variance, occasional large importance weights can still cause instability": I think what is meant is using *large* values of lambda.
- Above eq. (6) mention that the squared error is used.
- Missing a "t" subscript at the beginning of eq. (9)?
- It was hard to understand the stochastic duelling networks. Please rephrase this part.
- Please clarify this sentence "To compare different agents, we adopt as our metric the median of the human normalized score over all 57 games."
- Figure 2 (Bottom): Please add label to vertical axes.
review_title: Review

final_score: rating: null
review_content: - I thought one of the conclusions of the Mnih et al. paper on A3C was that experience replay was not required with Actor-Critic (assuming parallel agents). You seem to draw a different conclusion.  What should we think of all this?

- The role of the average policy network seems to be to stabilize the per-step changes in the policy.  Why isn’t gradient clipping sufficient for this?

- Fig.1: Why don’t you include confidence intervals (as you do in Fig.2)?

- Fig.2:  How does ACER compare to Generalized Advantage Learning (Schulman et al. 2016)?

review_title: Questions

final_score: rating: null
review_content: I appreciate that you evaluate your approach on both Atari games and continuous control problems. Your evaluation seems to be very thorough.
review_title: Testing on Atari games and continuous control problems

Meta-review: pros:
 - set of contributions leading to SOTA for sample complexity wrt Atari (discrete) and continuous domain problems
 - significant experimental analysis
 - long all-in-one paper
 
 cons:
 - builds on existing ideas, although ablation analysis shows each to be essential
 - long paper
 
The PCs believe this paper will be a good contribution to the conference track.
Decision: Accept (Poster)