Venue: ICLR 2017 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper proposed a new policy gradient method that uses the Taylor expansion of a critic as the control variate to reduce the variance in gradient estimation. The key idea is that the critic can be learned in an off-policy manner so that it is more sample efficient. Although the algorithm structure is similar to actor-critic, the critic information is “truncated” in a proper manner to reduce the variance in policy gradient. The proposed methods are evaluated on OpenAI Gym’s MuJoCo domains. Q-Prop is shown to produce more stable performance compared to DDPG and has higher sample efficiency than TRPO. 

The stability of off-policy TD learning for the critic is not guaranteed. Do the authors observe instability of it in the experiment? As the authors stated in the paper, the critic does not need to approximate the actual value function very well as long as it is correlated with \hat{A}. In the two adaptive Q-Prop schemes, the authors apply some tricks (conservative and aggressive adaptation) to control the possible unreliable estimate of the critic. This could be another evidence that the off-policy critic is not reliable. The authors may need to comment more on this point. Especially, it will be useful if the authors could show/justify that by such a design Q-Prop is robust against unreliable critic estimate. 

The authors seem to indicate that the advantage of Q-Prop over DDPG is in its insensitivity to hyperparameters. In Figure 3(a), the authors show that DDPG is sensitive to hyperparameters. However, the sensitivity of Q-Prop to the same hyperparameter is not shown. Experiments in the paper show that Q-Prop has advantage over TRPO in sample complexity. However, not much experiments are shown to justify the advantage of Q-Prop over DDPG. This is important because Table 1 shows that TR-c-Q-Prop needs significantly more samples than DDPG on Hopper, HalfCheetah and Swimmer. Any comment on that?

review_title: Off-policy TD learning of the critic

final_score: rating: null
review_content: No questions
review_title: No questions

final_score: rating: 7: Good paper, accept
review_content: **Edit: Based on the discussion below, my main problem (#2) was not correct. I have changed my overall rating from a 3 to a 7**

This paper makes a fascinating observation: one can introduce an action-dependent baseline (control variate) into REINFORCE, which introduces bias, and then include a correction term to remove the bias. The variance of the correction term is low relative to the REINFORCE update and the action-dependent baseline, and so this results in benefits.

However, the paper is poorly executed. Below I list my concerns.

1. The paper tries to distinguish between "policy gradient" methods and "actor critic" methods by defining them in a non-standard way. Specifically, when this paper says "policy gradient" it means REINFORCE. Historically, the two have meant different things: some policy gradient algorithms are actor-critics (e.g., Degris et al's INAC algorithm) while others are not (e.g. REINFORCE).

2. The proposed Q-Prop algorithm includes many interesting design choices that make in unclear what the real source of improved performance is. Is the improved performance due to the use of the action-dependent control variate? Would the same setup but using a state-value baseline still perform just as well? Are the performance benefits due to the use of an off-policy advantage estimation algorithm, GAE(lambda)? Or, would performance have been similar with an on-policy advantage estimation algorithm? What about if a different off-policy advantage estimation algorithm was used, like Retrace(lambda), GTD2, ETD, or WIS-LSTD? Or, is the improved performance due to the use of a replay buffer?

Comparisons are not performed between variants of Q-Prop that show the importances of these different components. Rather the authors opt to show better performance on a benchmark task. I find this to be non-scientific, and more of a paper showing a feat of engineering (by combining many different ideas) than it is a research paper that studies the details of which parts of Q-Prop make it work well. For example, after reading this paper, it is not clear whether having the action-dependent baseline (or using the first order Taylor approximation for the baseline) is beneficial or not - it could be that the strong performance comes from GAE(lambda) or the use of a replay buffer. At the very least I would have expected comparisons to Q-Prop using a state-value baseline (which would then be a variant of REINFORCE using off-policy data and a replay buffer, and which would show whether the action-dependent baseline is important).

3. There is a fair amount of discussion about unbiased policy gradient algorithms, which is not accurate. Most policy gradient algorithms are biased, and making them unbiased tends to hurt performance. This is discussed in the paper "Bias in Natural Actor-Critic Algorithms", which applies to non-natural algorithms as well. Also, I suspect that the use of GAE(lambda) results in the exact sort of bias discussed in that paper, even when lambda=1. As a result, Q-Prop may act more like an average reward method than expected. This should be discussed.

4. The proposed algorithm can be applied to deep architectures, just as most linear-time policy gradient algorithms can. However, it does not have to be applied to deep architectures. The emphasis on "deep" therefore seems to detract from the core ideas of the paper.

5. The paper repeatedly says that importance sampling based methods result in high variance. This ignores weighted importance sampling methods that have very low variance. A good example of this is Mahmood et al's WIS-LSTD algorithm. WIS-LSTD has high computational complexity, so it would only be compared to on non-deep RL problems, of which there are plenty. Alternatively, algorithms like Retrace(lambda) have quite low variance since the likelihood ratios are never bigger than one. Others might argue that ETD algorithms are currently the most effective. The simple dismissal of these algorithms because the original importance sampling estimator proposed in 2000 has high variance is not sufficient.

6. The paper does not compare to natural actor-critic algorithms. Once the weights, w, have been computed, REINFORCE uses samples of states from the normalized discounted state distribution and samples of the corresponding returns to estimate the policy gradient. One of the main reasons Q-Prop should work better than REINFORCE is that it includes a control variate that reduces the variance of the policy gradient update after w has been computed. Now, compare this to natural policy gradient algorithms. Once the weights, w, have been computed (admittedly, using compatible features for the advantage estimation but any features for the state-value estimation) the resulting update is = w. That is, is has zero variance and does not require additional sampling. It is as though a perfect control variate was used. Furthermore, natural gradient algorithms can be applied to deep architectures. Degris et al's INAC algorithm is linear time. Desjardin et al's "natural neural networks" paper also discusses efficient implementations of natural gradients for neural networks. Dabney's Natural Temporal Difference algorithms have linear time variants that fit this paper's description of actor-critic algorithms.

To summarize, given the weights w, REINFORCE has high variance, and Q-Prop claims to reduce the variance of REINFORCE. However, natural policy gradient methods have zero variance given the weights w. So, what is the benefit of Q-Prop over natural gradient algorithms using off-policy value function estimation methods to estimate Q (or A)? That is, why should we expect Q-Prop to perform better than NAC-LSTD using GAE(lambda) with experience replay in place of LSTD?

7. Equation (2) is false. The right side is proportional to the left side, not equal to it. There is a (1-gamma) term missing. There are also other typos throughout (e.g., Q and A sometimes are missing their action arguments).


Although I have listed my concerns, I would like to re-iterate that I do find the idea of an action-dependent baseline fascinating. My problem with this paper is with its execution, not with the novelty, impact, or quality of the core idea.
review_title: A good idea, but not a research paper

final_score: rating: 7: Good paper, accept
review_content: The paper proposes using a first-order Taylor expansion as a control variate in policy gradient-style methods. Empirical results in dynamical control tasks suggest that this algorithm reduces the sample complexity, while the theoretical results presented suggest the algorithm is unbiased but of lower variance.
The use of control variates is very important and the present paper is an interesting approach in this direction. I am not fully convinced of the approach, because it is one of many possible, and the theoretical analysis relies on an approximation of the variance rather than exact calculations, which makes it less compelling. However, this paper is a step in the right direction so it is worth accepting. In the experiments, a few things need to be discussed further:
- What is the running time of the proposed approach? The computation of the extra terms required looks like it could be expensive. running time comparison in addition to sample comparison should be included
- The sensitivity to parameter settings of the proposed algorithm needs to be illustrated in separate graphs, since this is one of the main claims in the paper
- It would be nice to have a toy example included in which one can actually compute exact values and plot learning curves to compare more directly bias and variance. It would especially be nice to do this with a task that includes rare states, which is the case in which variance of other methods (eg importance sampling) really becomes significant.
review_title: An interesting approach for using control variables to improve stability of deep RL control

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper presents  a model-free policy gradient approach for reinforcement learning that combines on-policy updates with an off-policy critic. The hope is to learn continuous control in a sample-efficient fashion. The approach is validated on a number of low-dimensional continuous control tasks in a simulated environment.

The paper is very well written, easy to follow, and provides an adequate context with which to appreciate the contributions it brings. Although this reviewer is not an expert in this literature, the proposed approach appears novel. The Q-Prop estimator appears to be a general and useful method for policy learning, and the experimental validations provide adequate support for the claims of improved sample efficiency. The detailed derivations given in the Supplementary Materials are very useful.

I like the paper and I don’t have much to comment on. Perhaps a discussion of the following aspects would add to the depth:
1) comparison of the methods at a given computational cost, instead of by the number of episodes seen.
2) discussion of the limitations of the technique: are there situations where convergence is difficult

Possible typo: in equation (4), should we read $… + \gamma Q_w( …$ instead of $… + \gamma Q( …$ ?
If not, then what is Q() without subscript w?

review_title: Efficient Policy Gradient using a Critic 

Meta-review: This paper presents a nice contribution to the RL literature, finding an intermediate point between the high-variance (but unbiased) gradient estimates from policy optimization methods, and low(er)-variance (but biased) gradient estimates from off-policy actor-critic methods like DDPG. The basic idea (as I'm interpreting it, similar to one of the reviewers below), that that we can use an action-dependent baseline, based upon off-policy learning, to lower the variance of the gradient, assuming we also correct for it in the gradient computation. The experiments clearly show the benefit of the proposed approach. The work is a nice combination/unification of two prominent trends in RL (with the overarching goal of reducing sample complexity, which is of course crucial here), and I believe is absolutely worth accepting. The authors also did an excellent job responding to reviewer concerns and adjusting the manuscript to address any issues raised.
 
 Pros:
 + Nice contribution combining off-policy and on-policy methods, with a novel and compelling algorithm
 + Good evaluation on a wide variety of control tasks
 
 Cons:
 - Somewhat difficult to understand (the interpretation I give above is not quite how the paper is presented, though I believe they are equivalent), and the given presentation is somewhat dense at time
Decision: Accept (Oral)