Title: Sample Efficient Actor-Critic with  Experience Replay
Authors: Ziyu Wang
Victor Bapst
Nicolas Heess
Volodymyr Mnih
Remi Munos
Koray Kavukcuoglu
Nando de Freitas
Venue: ICLR 2017
Keywords: Deep learning
Reinforcement Learning

Abstract: This paper presents an actor-critic deep reinforcement learning agent with experience replay that is stable, sample efficient, and performs remarkably well on challenging environments, including the discrete 57-game Atari domain and several continuous control problems. To achieve this, the paper introduces several innovations, including truncated importance sampling with bias correction, stochastic dueling network architectures, and a new trust region policy optimization method.

TL;DR: Prepared for ICLR 2017.

Review discussion:
I appreciate that you evaluate your approach on both Atari games and continuous control problems. Your evaluation seems to be very thorough.
- I thought one of the conclusions of the Mnih et al. paper on A3C was that experience replay was not required with Actor-Critic (assuming parallel agents). You seem to draw a different conclusion.  What should we think of all this?
- The role of the average policy network seems to be to stabilize the per-step changes in the policy.  Why isn’t gradient clipping sufficient for this?
- Fig.1: Why don’t you include confidence intervals (as you do in Fig.2)?
- Fig.2:  How does ACER compare to Generalized Advantage Learning (Schulman et al. 2016)?
Thanks very much for your questions.
You are correct that A3C does not require experience replay. The argument the authors made in the A3C paper is such that A3C can be stable without experience replay (unlike in the case of DQN). By doing away with replay, however, A3C suffers from poor sample-efficiency. The aim of our submission is to increase the sample-efficiency of actor critic algorithms by using experience replay.
Therefore, I don't think we are making a conflicting conclusion.
The functionality of the average policy network is indeed to stabilize training and I would argue that gradient clipping cannot accomplish this task by itself. The reason is that the norm (L_1, L_2, or L_inf) of a gradient update does not necessarily correspond to the amount of change in the policy (e.g. measured by KL).
Therefore, even a gradient update that is small in norm can change the policy drastically. Our proposed trust region approach works directly in the space of the policies and does not suffer from the same problem.
We did not include any confidence intervals in Fig 1 because we only used fixed hyper-parameters and one seed.
Generalized Advantage Estimation (GAE) is orthogonal the contribution of this paper. GAE, in the on-policy case, is equivalent to using lambda weighted returns to estimate the advantage for policy learning. It, however, does not handle off-policy learning which is the main focus our submission.
In addition, since ACER uses either Retrace(\lambda) of Q(\lambda) with off-policy correction to estimate the advantage, it already uses \lambda weighted returns to estimate the advantage (although we use lambda=1 in our paper for simplicity).
I hope this answers your questions. Thanks again!
The paper looks at several innovations for deep RL, and evaluates their effect on solving games in the Atari domain.  The paper reads a bit like a laundry list of the researcher’s latest tricks.  It is written clearly enough, but lacks a compelling message.  I expect the work will be interesting to people already implementing deep RL methods, but will probably not get much attention from the broader community.
The claims on p.1 suggest the approach is stable and sample efficience, and so I expected to see some theoretical analysis with respect to these properties.
But this is an empirical claim; it would help to clarify that in the abstract.
The proposed innovations are based on sound methods.  It is particularly nice to see the same approach working for both discrete and continuous domains.
The paper has reasonably complete empirical results. It would be nice to see confidence intervals on more of the plots.
Also, the results don’t really tease apart the effect of each of the various innovations, so it’s harder to understand the impact of each piece and to really get intuition, for example about why ACER outperforms A3C.
Also, it wasn’t clear to me why you only get matching results on discrete tasks, but get state-of-the-art on continuous tasks.
The paper has good coverage of the related literature. It is nice to see this work draw more attention to Retrace, including the theoretical characterization in Sec.7.
Thank you for your review. Please refer to our general reply titled “ablations and why each ingredient is an important contribution on its own”. It should clarify that instead of a “laundry list”, what we are proposing is actually a set of methods that all play together to achieve a single goal: a well founded and stable RL agent with K-step backups. We also believe that building one of the most powerful, stable, general, off-policy, core deep RL agents in existence makes for a compelling message.
We will clarify that our claim on sample efficiency and stability is largely based on empirical evidence, thanks for pointing this out.
The ideas underlying ACER are carefully chosen to obtain an off-policy actor-critic policy gradient algorithm that allows bias-corrected and low-variance multi-step updates, that can be applied both to discrete and continuous action spaces, and which is stable and sample-efficient in practice. All ideas in the paper are required to achieve this goal. (Please see the ablation analysis in the updated paper.) Furthermore, we believe that several innovations of ACER have much broader use cases:
Our proposed trust region technique, for example, can be easily adapted to be used in supervised learning as well as training variational auto-encoders or GANs.
Stochastic dueling networks are the first method that enables Retrace like methods to be used in the continuous domain.
Stochastic dueling networks also allow us to learn about state values in the continuous domain without having to resort to importance sampling.
We have added Figure 4 demonstrating the value of each component. We also demonstrate the importance of the trust region constraint for both continuous and Atari domains (for Atari notice the gap between solid, with trust region, vs. dashed lines, without trust region, in Figure 1) . The effect of Retrace is highlighted by the comparison between Trust-TIS and ACER (see Figure 3).
In terms of achieved median scores, one version of ACER is also state-of-the-art in discrete spaces (see general discussion). ACER works extremely well in both discrete and continuous action spaces. Considering ACER’s performance in both discrete and continuous environments, the advantage that ACER holds over A3C is similar in both cases. In Figure 1, we can see that ACER does better than A3C, in the discrete case, when we have a replay ratio of 1. In the continuous domains, A3C suffers from higher variance.
This paper introduces an actor-critic deep RL approach with experience replay, which combines truncated importance sampling and trust region policy optimization. The paper also proposes a new method called stochastic duelling networks to estimate the critic for continuous action spaces. The method is applied to Atari games and continuous control problems, where it yields performance comparable to state-of-the-art methods.
As mentioned in the beginning of the paper, the main contributions of this work lies in combining 1) truncated importance sampling with retrace, 2) trust region policy optimization, and 3) stochastic duelling networks. These improvements work well and may be beneficial to future work in RL.
However, each improvement appears to be quite incremental.
Moreover, the ACER framework seems much more complex and fragile to implement compared to the standard deep q-learning with prioritized replay (which appears to perform just as well on Atari games).
So for the Atari domain, I would still put my money on prioritized replay due to its simplicity.
Thirdly, improving sample efficiency for deep RL is a laudable goal, but really this goal should be pursued in a problem setting where sample efficiency is important. Unfortunately, the paper only evaluates sample efficiency in the Atari and continuous control tasks domain; two domains where sample efficiency is not important.
Thus, it is not clear that the proposed method ACER will generalize to problems where we really care about sample efficiency.
Some technical aspects which need clarifications:
- For Retrace, I assume that you compute recursively $Q^{ret}$ starting from the end of each trajectory? Please comment on this.
- It's not clear to me how to derive eq. (7). Is an approximation (double tilde) sign missing?
- In section 3.1 the paper argued that $Q^{ret}$ gives a lower-variance estimate of the action-value function.
Then why not use it in eq. (8) for the bias correction term?
- The paper states that it uses a replay memory of 50000 frames, so that across threads it is comparable in size to previous work.
However, for each thread this is much smaller compared to earlier experiments on Atari games.
For example, one million experience replay transitions were used in the paper "Prioritized Experience Replay" by Schaul et al. This may have a huge impact on performance of the models (both for ACER and for the competing models). In order to properly assess the improvements of ACER over previous work, the authors need to also experiment with larger experience replay memories.
Other comments:
- Please move Section 7 to the appendix.
- "Moreover, when using small values of lambda to reduce variance, occasional large importance weights can still cause instability": I think what is meant is using *large* values of lambda.
- Above eq. (6) mention that the squared error is used.
- Missing a "t" subscript at the beginning of eq. (9)?
- It was hard to understand the stochastic duelling networks. Please rephrase this part.
- Please clarify this sentence "To compare different agents, we adopt as our metric the median of the human normalized score over all 57 games."
- Figure 2 (Bottom): Please add label to vertical axes.
Thank you for your detailed review and comments. Please refer to our general reply titled “ablations and why each ingredient is an important contribution on its own”.
In addition to the ablations shown in Figure 4 of the revised paper,  which show significant gains in performance for each proposed component, we would like to reply your your review in more detail.
Regarding contributions: Although the ACER framework builds on some ideas from the literature (asynchronous policy gradient; experience replay; Retrace) it also makes several significant contributions:
- We introduce an novel approximate trust region method that is effective yet computationally lightweight. This technique can be easily adapted to be used in supervised learning as well as training variational auto-encoders or GANs.
- We introduce a novel truncation/correction procedure which truncates the importance sampling ratio in the likelihood ratio policy gradient estimate in order to reduce its variance, and correct for the bias (introduced by the truncation) using the estimated Q-values.
- We introduce stochastic dueling networks which are the first method enabling Retrace like methods to be used in the continuous domain.
- Finally ACER integrates these ideas into a PG framework that works for both continuous and discrete action spaces. ACER also bridges the gap between highly data-efficient ER and Q-function based algorithms  (DQN, DPG) on the one hand and scalable but data-inefficient and in some cases less stable (raw A3C) high-throughput algorithms on the other, allowing to interpolate between these two extremes and also improving the stability of the latter.
Regarding complexity: We would argue that the ACER framework is neither more complex nor fragile than the DQN framework or many of the improvements to DQN.
Regarding the implementation: all three components of ACER require very little change in code given A3C and replay memory (< 30 lines of torch code).
In contrast, prioritized replay, although conceptually simple, is much more demanding to implement due to the need to sort and sample (not uniformly) from a large replay memory.
Regarding stability: ACER is also not fragile which is evident from the fact that we use one single hyper-parameter to play all Atari games. For the continuous control tasks, the sensitivity analysis is in the appendix.
The innovations we advance in ACER are orthogonal to prioritized replay and we believe ACER can benefit tremendously from prioritized replay when properly adapted.
Furthermore, ACER naturally applies to both discrete and continuous problems whereas DQN with prioritized replay has only been demonstrated to perform well on discrete problems.
Usefulness of sample complexity for evaluation: Firstly, we would like to point out that we evaluate ACER not just in terms of sample complexity, but demonstrate, in the context of the continuous control tasks, how components of the algorithm are also improving robustness and asymptotic performance.
Secondly, (although not directly tested in the paper) off-policy learning also increases the flexibility e.g. the use of exploration policies different from the target policy.
Thirdly, continuous control in the context of robotics is certainly an area where sample complexity is of great importance.
Finally, even for entirely simulated environments sample efficiency may matter from a computational perspective.
For instance, for visually complex environments rendering time may be so significant that sample reuse via replay becomes advantageous.
Answers to technical aspects which need clarifications:
We do calculate Q^{ret} from the end of the trajectory (or the end of a truncated trajectory).
Eqn (7) is describing the true policy gradient (not an approximation).
Therefore the double tilde is not present. To derive Eqn. (7), one has to truncate the importance ratio and compensate. It should help to notice that \E_{a\sim\pi} means sampling from the target policy (\pi) whereas \E_{a_t} means sampling from the behavior distribution (\mu). The change of variable from one to the other requires introducing an importance sampling ratio \rho. We will make this point clearer.
For the bias correction term, Q^{ret} is not available in practice. This is because we did not necessarily take the actions in the bias correction term, so we cannot use the trajectories to approximate Q^{ret}.
That is, the trajectories are generated with the actions taken.
We did try 100000 frames for the replay memory and it did not seem to have a significant effect. We will mention this in the paper. Having 1Mil frames per thread, unfortunately, is currently not feasible due to memory constraints.
Answers to other comments:
- Please move Section 7 to the appendix.
* We will consider moving it in the final version of the paper. Thanks for the comment.
- "Moreover, when using small values of lambda to reduce variance, occasional large importance weights can still cause instability": I think what is meant is using *large* values of lambda.
* We do mean importance weights which can be much larger than 1 in which case we would have high variance.  Lambda is a value of our choosing.
- Above eq. (6) mention that the squared error is used.
* We have fixed it. Thanks.
- Missing a "t" subscript at the beginning of eq. (9)?
* We have fixed it. Thanks.
- It was hard to understand the stochastic duelling networks. Please rephrase this part.
* We have added a figure with a diagram of the stochastic dueling network. Hopefully this helps to clarify the idea.
- Please clarify this sentence "To compare different agents, we adopt as our metric the median of the human normalized score over all 57 games."
* The paragraph following this sentence explains it in more detail.
- Figure 2 (Bottom): Please add label to vertical axes.
*We have fixed it. Thanks.
This paper studies the off-policy learning of actor-critic with experience replay. This is an important and challenging problem in order to improve the sample efficiency of the reinforcement learning algorithms. The paper attacks the problem by introducing a new way to truncate importance weight, a modified trust region optimization, and by combining retrace method. The combination of the above techniques performs well on Atari and MuJoCo in terms of improving sample efficiency. My main comment is how does each of the technique contribute to the performance gain? If some experiments could be carried out to evaluate the separate gains from these tricks, it would be helpful.
Thank you for your review on Xmas day! Please refer to our general reply titled “ablations and why each ingredient is an important contribution on its own”.
We have added a more comprehensive ablation analysis that shows the effect of each major component of ACER (Figure 4). These experiments show that the major components of ACER are all essential to achieve its excellent performance in both discrete and continuous settings. Part of the contribution of ACER is that it brings carefully designed components together to obtain a single off-policy actor-critic policy gradient algorithm that allows bias-corrected and low-variance multi-step updates, that can be applied to both discrete and continuous action spaces, and that is stable and sample-efficient in practice.
However, several of the components are important contributions on their own as Figure 4 shows. Thanks for requesting it.
We chose to write a single comprehensive and solid 20-page manuscript, instead of several papers with one good contribution (one for off-policy policy gradients with retrace, one for the new efficient way of doing trust region, one for stability and developing the theory of truncated importance sampling with bias correction, etc. ). Our excellent results would have enabled to do this. We did the right thing and hope not to be penalized for this.
Thanks for carrying out new ablation analysis. The results addressed my concern and are convincing.
Therefore, I bump up the score.
Thank you!
We thank the three reviewers. The one common concern is ablations. This paper proposes several new ideas, and then goes on to combine these ideas.
To answer the reviewers concerns about ablations, we added a new figure (Figure 4). This is an extremely important figure and we urge the reviewers and readers to consult it as it should answer any concerns and highlight the value of the many contributions made in this paper. The figure shows that each ingredient (Retrace/Q-lambda with off-policy correction, stochastic dueling nets, and the NEW trust region method) on its own leads to a massive improvement.
Likewise, truncation with bias correction plays an important role for large action spaces (control of humanoid). This figure indicates that this paper is not about making 4 small contributions and combining them.
Rather it is about making 4 important contributions, which are all essential to obtain a stable, scalable, general, off-policy actor critic. Attaining this has been a holy grail, and this paper shows how to do it.
Given our good results, we could easily have written several papers; one for each contribution.
Instead, we chose to do the honest thing and write a single solid 20-page paper aimed at truly building powerful deep RL agents for both continuous and discrete action spaces. The paper also presents novel theoretical results for RL, and a very comprehensive experimental study.
We did not want to claim state-of-the-art on Atari because this often depends on how one chooses to measure what should be state-of-the-art (eg sample complexity, highest median, highest mean, etc.).
But clearly, in terms of median, ACER with 1 replay achieves a higher median score that any previously reported result. Note that this result is not just for a few games, but for the entire set of 57 games. The UNREAL agent submitted to this conference is the only method we know that achieves a higher median, but it does so by adding auxiliary tasks to A3C and massive hyper-parameter sweeps. We could also add auxiliary tasks to ACER and do hyper-parameter sweeps to further improve it, but this is left for future work as we wanted to focus on designing a powerful core RL agent.
We hope this reply and in particular the ablations clearly answer your concerns. With 3 6’s this thorough paper will be rejected despite the several novel contributions it makes, new theoretical analysis, and excellent results on a comprehensive set of tasks. We hope you take the ablations and this reply into consideration to choose your final scores.
Dear reviewers, we would really appreciate it if you can take a look at the paper again in light of our replies, the updated paper, and the comments from Xi Chen. Thanks very much for your time!

Meta-review:
role: PC
data: comment: pros:
 - set of contributions leading to SOTA for sample complexity wrt Atari (discrete) and continuous domain problems
 - significant experimental analysis
 - long all-in-one paper
 
 cons:
 - builds on existing ideas, although ablation analysis shows each to be essential
 - long paper
 
The PCs believe this paper will be a good contribution to the conference track.

Decision: Accept (Poster)