Title: Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Critic
Authors: Shixiang Gu
Timothy Lillicrap
Zoubin Ghahramani
Richard E. Turner
Sergey Levine
Venue: ICLR 2017
Keywords: Deep learning
Reinforcement Learning

Abstract: Model-free deep reinforcement learning (RL) methods have been successful in a wide variety of simulated domains. However, a major obstacle facing deep RL in the real world is their high sample complexity. Batch policy gradient methods offer stable learning, but at the cost of high variance, which often requires large batches. TD-style methods, such as off-policy actor-critic and Q-learning, are more sample-efficient but biased, and often require costly hyperparameter sweeps to stabilize. In this work, we aim to develop methods that combine the stability of policy gradients with the efficiency of off-policy RL. We present Q-Prop, a policy gradient method that uses a Taylor expansion of the off-policy critic as a control variate. Q-Prop is both sample efficient and stable, and effectively combines the benefits of on-policy and off-policy methods. We analyze the connection between Q-Prop and existing model-free algorithms, and use control variate theory to derive two variants of Q-Prop with conservative and aggressive adaptation. We show that conservative Q-Prop provides substantial gains in sample efficiency over trust region policy optimization (TRPO) with generalized advantage estimation (GAE), and improves stability over deep deterministic policy gradient (DDPG), the state-of-the-art on-policy and off-policy methods, on OpenAI Gym's MuJoCo continuous control environments.

TL;DR: We propose Q-Prop, a novel policy gradient method with an off-policy critic as control variate, that is more sample efficient than TRPO-GAE and more stable than DDPG, the state-of-the-art on-policy and off-policy methods.

Review discussion:
This paper proposed a new policy gradient method that uses the Taylor expansion of a critic as the control variate to reduce the variance in gradient estimation. The key idea is that the critic can be learned in an off-policy manner so that it is more sample efficient.
Although the algorithm structure is similar to actor-critic, the critic information is “truncated” in a proper manner to reduce the variance in policy gradient. The proposed methods are evaluated on OpenAI Gym’s MuJoCo domains. Q-Prop is shown to produce more stable performance compared to DDPG and has higher sample efficiency than TRPO.
The stability of off-policy TD learning for the critic is not guaranteed. Do the authors observe instability of it in the experiment? As the authors stated in the paper, the critic does not need to approximate the actual value function very well as long as it is correlated with \hat{A}. In the two adaptive Q-Prop schemes, the authors apply some tricks (conservative and aggressive adaptation) to control the possible unreliable estimate of the critic. This could be another evidence that the off-policy critic is not reliable. The authors may need to comment more on this point. Especially, it will be useful if the authors could show/justify that by such a design Q-Prop is robust against unreliable critic estimate.
The authors seem to indicate that the advantage of Q-Prop over DDPG is in its insensitivity to hyperparameters. In Figure 3(a), the authors show that DDPG is sensitive to hyperparameters.
However, the sensitivity of Q-Prop to the same hyperparameter is not shown. Experiments in the paper show that Q-Prop has advantage over TRPO in sample complexity.
However, not much experiments are shown to justify the advantage of Q-Prop over DDPG. This is important because Table 1 shows that TR-c-Q-Prop needs significantly more samples than DDPG on Hopper, HalfCheetah and Swimmer. Any comment on that?
Thank you for your review and constructive feedback!
Figure 2.a shows that on HalfCheetah-v1 the conservative variant of Q-Prop (TR-c-Q-Prop) learns more stably than the standard (TR-Q-Prop) and aggressive (TR-a-Q-Prop) variants, and empirically appears more robust toward unreliable critic estimate. This is expected from the derivation, since the conservative variant effectively turns off Q-Prop and back-off to TRPO for states that are likely to have bad critic estimates. We also updated Section 3.4 second paragraph to include discussions on the robustness to bad critics.
We performed hyperparameter search on both Q-Prop and DDPG as detailed in Appendix D and showing that the best attainable performance is more stable for Q-Prop than DDPG as shown in Table 1. Our claim is that Q-Prop is more robust to bad critics and less sensitive to hyperparameters than DDPG. In Table 1, it is important to weigh significances of task results based on task difficulties.
While DDPG could solve tasks like HalfCheetah and Swimmer (which have 17 and 8 state dimensions and 6 and 2 action dimensions respectively), Figure 3. b and Table 1 show that Q-Prop achieves significantly better performance within the same number of episodes on the much harder Humanoid task (376 state dimensions and 17 action dimension). Note that Q-Prop was able to solve all tasks as well as TRPO, though sometimes slightly slower than DDPG, while DDPG was unable to achieve good performance on Humanoid. Empirically, we observe that the harder the tasks become (e. g. 2D HalfCheetah -> 3D Humanoid, see curves comparing TR-c-Q-Prop and DDPG in Figures 3.
a, 3. b and Table 1), the benefit of Q-prop over purely off-policy method like DDPG becomes more obvious. A crucial point about DDPG from our experiment is that given a reasonable hyperparameter search done in Appendix D for DDPG, DDPG couldn’t find good solutions for some of the harder problems. This practically makes DDPG difficult to be applied for solving hard problems, even though it sometimes can solve simpler tasks better than TRPO and Q-Prop.
We also ran additional experiments to validate the hyperparameter sensitivity further, as available in https://docs.google.com/document/d/1ow_BIrKYt11r4BXMbM7w9qBxbedhedKrBrvMhnXm7nI/edit?usp=sharing. It builds on the discussion on Figure 3.a, further demonstrating that Q-Prop is less sensitive to reward scales than DDPG.
No questions
This paper presents  a model-free policy gradient approach for reinforcement learning that combines on-policy updates with an off-policy critic. The hope is to learn continuous control in a sample-efficient fashion. The approach is validated on a number of low-dimensional continuous control tasks in a simulated environment.
The paper is very well written, easy to follow, and provides an adequate context with which to appreciate the contributions it brings.
Although this reviewer is not an expert in this literature, the proposed approach appears novel. The Q-Prop estimator appears to be a general and useful method for policy learning, and the experimental validations provide adequate support for the claims of improved sample efficiency. The detailed derivations given in the Supplementary Materials are very useful.
I like the paper and I don’t have much to comment on. Perhaps a discussion of the following aspects would add to the depth:
1) comparison of the methods at a given computational cost, instead of by the number of episodes seen.
2) discussion of the limitations of the technique: are there situations where convergence is difficult
Possible typo: in equation (4), should we read $… + \gamma Q_w( …$ instead of $… + \gamma Q( …$ ?
If not, then what is Q() without subscript w?
Thank you for your review and positive feedback!
1) compute time: We expanded current discussion on computation time in Section 3.4. The running time of Q-Prop is effectively time(TRPO) + time(DDPG)/2, if we use the same parameter update per step of experience as DDPG (i.e. one update per one step). If the simulator is very fast, time(DDPG)/2 >> time(TRPO), so effectively the run-time per episode of Q-Prop is about half of DDPG and slower than TRPO.
However, for applications where we care about sample-efficiency, the experience collection is often the bottleneck, and in such cases, the off-policy critic learning and the on-policy data collection and policy update can be parallelized, and will run at about the same speed as both TRPO and DDPG.
2) limitations of this technique: We added Section 3.4 to discuss the main limitations.
We also clarified the confusion in Eq (4). Q without subscript represents the target network.
The paper proposes using a first-order Taylor expansion as a control variate in policy gradient-style methods. Empirical results in dynamical control tasks suggest that this algorithm reduces the sample complexity, while the theoretical results presented suggest the algorithm is unbiased but of lower variance.
The use of control variates is very important and the present paper is an interesting approach in this direction. I am not fully convinced of the approach, because it is one of many possible, and the theoretical analysis relies on an approximation of the variance rather than exact calculations, which makes it less compelling.
However, this paper is a step in the right direction so it is worth accepting. In the experiments, a few things need to be discussed further:
- What is the running time of the proposed approach? The computation of the extra terms required looks like it could be expensive. running time comparison in addition to sample comparison should be included
- The sensitivity to parameter settings of the proposed algorithm needs to be illustrated in separate graphs, since this is one of the main claims in the paper
- It would be nice to have a toy example included in which one can actually compute exact values and plot learning curves to compare more directly bias and variance. It would especially be nice to do this with a task that includes rare states, which is the case in which variance of other methods (eg importance sampling) really becomes significant.
Thank you for your detailed reviews and pointing to aspects of the paper that can be improved!
- compute time: We expanded current discussion on computation time in Section 3.4. The running time of Q-Prop is effectively time(TRPO) + time(DDPG)/2, if we use the same parameter update per step of experience as DDPG (i.e. one update per one step). If the simulator is very fast, time(DDPG)/2 >> time(TRPO), so effectively the run-time per episode of Q-Prop is about half of DDPG and slower than TRPO.
However, for applications where we care about sample-efficiency, the experience collection is the bottleneck, and in such cases, the off-policy critic learning and the on-policy data collection and policy update can be parallelized, and will run at about the same speed as both TRPO and DDPG.
- sensitivity experiment: We ran additional experiments to validate the hyperparameter sensitivity further, as available in https://docs.google.com/document/d/1ow_BIrKYt11r4BXMbM7w9qBxbedhedKrBrvMhnXm7nI/edit?usp=sharing. It builds on the discussion on Figure 3.a, further demonstrating that Q-Prop is less sensitive to reward scales than DDPG.
**Edit: Based on the discussion below, my main problem (#2) was not correct. I have changed my overall rating from a 3 to a 7**
This paper makes a fascinating observation: one can introduce an action-dependent baseline (control variate) into REINFORCE, which introduces bias, and then include a correction term to remove the bias. The variance of the correction term is low relative to the REINFORCE update and the action-dependent baseline, and so this results in benefits.
However, the paper is poorly executed. Below I list my concerns.
1. The paper tries to distinguish between "policy gradient" methods and "actor critic" methods by defining them in a non-standard way.
Specifically, when this paper says "policy gradient" it means REINFORCE. Historically, the two have meant different things: some policy gradient algorithms are actor-critics (e.g., Degris et al's INAC algorithm) while others are not (e.g. REINFORCE).
2. The proposed Q-Prop algorithm includes many interesting design choices that make in unclear what the real source of improved performance is. Is the improved performance due to the use of the action-dependent control variate? Would the same setup but using a state-value baseline still perform just as well? Are the performance benefits due to the use of an off-policy advantage estimation algorithm, GAE(lambda)? Or, would performance have been similar with an on-policy advantage estimation algorithm? What about if a different off-policy advantage estimation algorithm was used, like Retrace(lambda), GTD2, ETD, or WIS-LSTD? Or, is the improved performance due to the use of a replay buffer?
Comparisons are not performed between variants of Q-Prop that show the importances of these different components.
Rather the authors opt to show better performance on a benchmark task. I find this to be non-scientific, and more of a paper showing a feat of engineering (by combining many different ideas) than it is a research paper that studies the details of which parts of Q-Prop make it work well.
For example, after reading this paper, it is not clear whether having the action-dependent baseline (or using the first order Taylor approximation for the baseline) is beneficial or not - it could be that the strong performance comes from GAE(lambda) or the use of a replay buffer. At the very least I would have expected comparisons to Q-Prop using a state-value baseline (which would then be a variant of REINFORCE using off-policy data and a replay buffer, and which would show whether the action-dependent baseline is important).
3. There is a fair amount of discussion about unbiased policy gradient algorithms, which is not accurate. Most policy gradient algorithms are biased, and making them unbiased tends to hurt performance. This is discussed in the paper "Bias in Natural Actor-Critic Algorithms", which applies to non-natural algorithms as well.
Also, I suspect that the use of GAE(lambda) results in the exact sort of bias discussed in that paper, even when lambda=1.
As a result, Q-Prop may act more like an average reward method than expected. This should be discussed.
4. The proposed algorithm can be applied to deep architectures, just as most linear-time policy gradient algorithms can.
However, it does not have to be applied to deep architectures. The emphasis on "deep" therefore seems to detract from the core ideas of the paper.
5. The paper repeatedly says that importance sampling based methods result in high variance. This ignores weighted importance sampling methods that have very low variance. A good example of this is Mahmood et al's WIS-LSTD algorithm. WIS-LSTD has high computational complexity, so it would only be compared to on non-deep RL problems, of which there are plenty.
Alternatively, algorithms like Retrace(lambda) have quite low variance since the likelihood ratios are never bigger than one. Others might argue that ETD algorithms are currently the most effective. The simple dismissal of these algorithms because the original importance sampling estimator proposed in 2000 has high variance is not sufficient.
6. The paper does not compare to natural actor-critic algorithms. Once the weights, w, have been computed, REINFORCE uses samples of states from the normalized discounted state distribution and samples of the corresponding returns to estimate the policy gradient. One of the main reasons Q-Prop should work better than REINFORCE is that it includes a control variate that reduces the variance of the policy gradient update after w has been computed. Now, compare this to natural policy gradient algorithms. Once the weights, w, have been computed (admittedly, using compatible features for the advantage estimation but any features for the state-value estimation) the resulting update is = w.
That is, is has zero variance and does not require additional sampling. It is as though a perfect control variate was used.
Furthermore, natural gradient algorithms can be applied to deep architectures. Degris et al's INAC algorithm is linear time. Desjardin et al's "natural neural networks" paper also discusses efficient implementations of natural gradients for neural networks. Dabney's Natural Temporal Difference algorithms have linear time variants that fit this paper's description of actor-critic algorithms.
To summarize, given the weights w, REINFORCE has high variance, and Q-Prop claims to reduce the variance of REINFORCE.
However, natural policy gradient methods have zero variance given the weights w.
So, what is the benefit of Q-Prop over natural gradient algorithms using off-policy value function estimation methods to estimate Q (or A)?
That is, why should we expect Q-Prop to perform better than NAC-LSTD using GAE(lambda) with experience replay in place of LSTD?
7. Equation (2) is false. The right side is proportional to the left side, not equal to it. There is a (1-gamma) term missing. There are also other typos throughout (e.g., Q and A sometimes are missing their action arguments).
Although I have listed my concerns, I would like to re-iterate that I do find the idea of an action-dependent baseline fascinating. My problem with this paper is with its execution, not with the novelty, impact, or quality of the core idea.
Thank you for valuable feedback. We hope we could engage in constructive discussions to fully clarify and address your concerns and questions. We have incorporated some of your suggestions into the paper, available on OpenReview. Below are our initial responses to your seven comments.
1. Given your feedback, we renamed Section 2.1 as “Monte Carlo policy gradient methods” and Section 2.2 as “policy gradient methods with function approximation”, added a discussion in Section 2, and clarified some references to “policy gradient” through the paper. We decided to use “Monte Carlo policy gradient” over “REINFORCE” because we would like it to cover a more general class of algorithms including REINFORCE/vanilla policy gradient, TRPO, etc. Please let us know if you have other suggestions.
2. We agree that a detailed evaluation of the components of the algorithm is important, and we endeavored to provide these in Figure 2.a (see curves comparing variants of Q-Prop) and Figure 3.a (see Q-Prop variants with and without trust-region, i.e. TR-c-Q-Prop and v-c-Q-Prop). Note, however, that Q-Prop is a general control variates approach that can be combined with a number of prior techniques, including TRPO-GAE and DDPG,  as well as natural gradient, standard REINFORCE, and standard (on-policy) actor-critic algorithms. We've clarified this in the second-to-last paragraph in Section 3.1.
While the particular implementation used in our experiments has a number of design choices, these are inherited from the prior methods. We believe it is therefore outside the scope of this paper (and outside of the conference paper length limits) to evaluate each of these decisions, but further discussion of these points can be found in the corresponding prior work: DQN (Mnih et. al. 2015), DDPG (Lillicrap et. al. 2016), TRPO-GAE (Schulman et. al., 2016). Below we respond to specific lines in this comment.
“Is the improved performance due to the use of the action-dependent control variate? Would the same setup but using a state-value baseline still perform just as well? Are the performance benefits due to the use of an off-policy advantage estimation algorithm, GAE(lambda)”
All our policy gradient/REINFORCE-based benchmark methods defined in section 5 first paragraph are implemented with GAE(lambda=0.97), which itself includes a state-value baseline, to give a fair comparison. Q-prop is implemented on top of these, so all improvements seen come from the use of our action-dependent baseline (see Figures 2.b, 3.a, 3.b and Table 1, e.g. compare Figure 2.b. "TRPO-5000" vs "TR-c-Q-Prop-5000", which indicates the benefit of Q-Prop over TRPO using the same exact estimator, with the only difference being the action-dependent baseline). In Figure 3.a, we also compared REINFORCE with and without Q-Prop (see “VPG” and “v-c-Q-Prop” curves). Note that REINFORCE here also uses the GAE state-dependent baseline, making this a reasonable REINFORCE comparison of action dependent vs state dependent baselines.
Lastly to clarify a detail, the original GAE paper uses on-policy advantage estimation and does not use replay buffer. This was in fact an important motivation for us to explore Q-Prop with a critic learned off-policy. We are in the process of running these experiments, but preliminary results suggest that an off-policy action-independent baseline (GAE with off-policy value function) performs worse than both an off-policy action-dependent baseline (Q-Prop) and an on-policy action-independent baseline (GAE).
“Or, would performance have been similar with an on-policy advantage estimation algorithm? What about if a different off-policy advantage estimation algorithm was used, like Retrace(lambda), GTD2, ETD, or WIS-LSTD? Or, is the improved performance due to the use of a replay buffer?”
We do not think Q-Prop with on-policy advantage estimation will provide much benefit over GAE because GAE already performs on-policy advantage estimation via an on-policy state-value function estimator. Using Q-Prop with an on-policy action-value function estimator is a possible alternative to investigate, which we are currently working to add to the experiments.
That said, we specifically focused on a simple off-policy action-value function fitting approach in this paper because we were aiming to improve sample-efficiency and wanted a simple demonstration to show using off-policy data in on-policy REINFORCE-style algorithms helps. We believe that good alternative off-policy advantage estimation algorithms are likely to do equally well or better, and we’ve added a discussion in the last paragraph of Section 4 on this. Comparing with all prior off-policy estimation approaches like Retrace(lambda), GTD2, ETD, or WIS-LSTD is interesting empirical work for the future;
however, we can only present a limited number of experimental results in a conference format, so we instead chose to implement Q-Prop on top of the current state-of-the-art deep RL policy gradient and actor-critic methods (TRPO and DDPG-style Q-function estimation), so as to cleanly illustrate the improvement provided by an action-dependent control variate obtained from off-policy data.
“Comparisons are not performed between variants of Q-Prop that show the importances of these different components.”
Figure 2.a is dedicated to comparing various variants of Q-Prop: no-Q-Prop (“TRPO”) vs. standard Q-Prop (“TR-Q-Prop”) vs conservative (“TR-c-Q-Prop”) vs aggressive (“TR-a-Q-Prop”). We felt this is more important than comparing the off-policy learning component with all previous variants in the limited space available in a conference paper. The results in Figure 2.a that TRPO-GAE+conservative Q-Prop performed most reliably and is indeed what we expected.
“For example, after reading this paper, it is not clear whether having the action-dependent baseline (or using the first order Taylor approximation for the baseline) is beneficial or not - it could be that the strong performance comes from GAE(lambda) or the use of a replay buffer.”
Again, Q-Prop is implemented on top of GAE, and comparing TR-c-Q-Prop curves with TRPO curves in Figures 2.b, 3.a, 3.b and Table 1 clearly show that Q-Prop further improves upon GAE.
To summarize, the Q-Prop method itself is very simple, and most implementation designs are directly inherited from prior state-of-the-art method papers (TRPO-GAE and DDPG). Q-Prop offers considerable flexibility on what on-policy and off-policy advantage learning techniques, and we have added discussion in the second-to-last paragraph in Section 3.1. Given that the current off-policy advantage learning is based on prior techniques (e.g. DDPG), more sophisticated methods (e.g. Retrace(lambda)) will likely lead to further improvements, and we’ve added a discussion of this to Section 4 last paragraph.
3. Thank you for pointing this out. It can be made unbiased either if gamma = 1 or if gamma^t multiplies the gradient at each time step, which is generally undesirable. This is general for most policy gradient/REINFORCE methods. We have included the discussion on this in Section 2.1 below Eq. 2, and appropriately qualified all claims of unbiasedness in the paper.
4. Thank you for pointing this out.
Indeed, the paper can be written differently to emphasize the point that this is not specific to deep neural networks.
However, our immediate goal in this paper is to make deep RL work better, and the design decisions in the algorithm are tailored to specifically addressed the difficulties of training deep architectures. Our aim in focusing on deep RL is to properly scope out the contribution of the paper. We are NOT claiming to propose a better RL algorithm for everything, because we don't have evidence for this, so while it's true that the contribution is not deep network specific, the evaluation is, and the focus of the paper reflects this limitation of the evaluation (due to the natural limitations of a fixed-size conference paper). We believe this scope is reasonable for ICLR, as evidenced by a number of other deep RL submissions that also propose improvements not specific to deep networks. It would be an interesting direction for future (e.g. an extended journal version) to evaluate the method in other, non-deep RL settings.
5. Thank you. We will update the discussion about importance sampling to include those references. If you would like to suggest a comparison to an importance sampling method that is suitable for large continuous state and action spaces, we would be happy to compare to it.
6. We already compare to a natural gradient method, namely TRPO, which is a variant of natural gradient with a specific step size rule (see Section 7 in Schulman et. al. 2015). On the tasks in our experiments, TRPO was shown to perform as well or better than standard natural policy gradient methods (see Figure 4 in Schulman et. al., 2015, and see Table 1 and Section 6 “TNPG and TRPO” in Duan et. al., 2016), and we therefore picked TRPO as the baseline in our experiments. In our paper, we show that adding Q-Prop further improves TRPO (see TR-c-Q-Prop curves with TRPO curves in Figures 2.b, 3.a, 3.b and Table 1).
Thus, we have compared our method with a natural gradient algorithm and shown improvement.
7. We fixed the problem by modifying rho_pi definition in Eq 2 and fixed the Q, A typos in Section 2.1 and \hat{Q} typo in Section 3.1. Thank you!
We appreciate that you find the idea of Q-Prop interesting. We made the decision to prioritize comparisons with the state-of-the-art deep RL methods because our priority in this paper is to improve upon deep RL methods;
however, we look forward to making more concrete analytical and empirical comparison with a broader class of RL techniques in future work, which we agree will be another valuable research contribution.
Thanks for the response.
2. This was my main concern, and perhaps I was wrong. What I missed was that your implementation of TRPO is *exactly* the same as Q-Prop if you use \eta_{t,e}=0. Can you confirm that this is correct? If so, then my primary concern about this paper was wrong. This implementation equivalence may be worth stating in the paper.
The other points are all relatively minor.
Also, is the source code for the experiments available somewhere?
Thank you for your quick response. Yes, the implementation of TRPO-GAE is exactly the same as Q-Prop if \eta_{t,e}=0. We will include the sentence about equivalence in the paper.
The codes are available: https://github.com/shaneshixiang/rllabplusplus. Q-Prop codes are directly written on top of the open-sourced TRPO-GAE TensorFlow implementation (e.g. sandbox/rocky/tf/algos/batch_opt.py).
Thank you for your feedback, and we're glad to hear that our response addressed your concerns. Since your answer indicates that you've reevaluated the paper based on our responses and additional experiments, we would appreciate a revision to the original score as well, since your current score is substantially lower than that of the other reviewers. And of course we would appreciate any other feedback if, e.g., we have not addressed all of your concerns in enough detail.
I have edited the original review to note that I now suggest the paper for acceptance.

Meta-review:
role: PC
data: comment: This paper presents a nice contribution to the RL literature, finding an intermediate point between the high-variance (but unbiased) gradient estimates from policy optimization methods, and low(er)-variance (but biased) gradient estimates from off-policy actor-critic methods like DDPG. The basic idea (as I'm interpreting it, similar to one of the reviewers below), that that we can use an action-dependent baseline, based upon off-policy learning, to lower the variance of the gradient, assuming we also correct for it in the gradient computation. The experiments clearly show the benefit of the proposed approach. The work is a nice combination/unification of two prominent trends in RL (with the overarching goal of reducing sample complexity, which is of course crucial here), and I believe is absolutely worth accepting. The authors also did an excellent job responding to reviewer concerns and adjusting the manuscript to address any issues raised.
 
 Pros:
 + Nice contribution combining off-policy and on-policy methods, with a novel and compelling algorithm
 + Good evaluation on a wide variety of control tasks
 
 Cons:
 - Somewhat difficult to understand (the interpretation I give above is not quite how the paper is presented, though I believe they are equivalent), and the given presentation is somewhat dense at time

Decision: Accept (Oral)