Venue: ICLR 2018 Workshop

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The paper presents an interesting critique of state-action baselines and how such baselines affect the variance of policy gradient estimators. The authors make two main contributions towards establishing efficacy of such baselines.

 (i) The first contribution is to decompose the variance of policy gradients into three components, $\sigma_t, \sigma_a, \sigma_s$. On continuous control tasks it is observed that $\sigama_t$ >> $\sigma_a$. While using an optimal state-action baseline would improve the \sigma_a term, the dominant $\sigma_t$ term still contributes to large variance.

(ii) When using GAE, the $\sigma_t$ term is reduced. However, the state-action and state based baselines produce similar variance estimates and these variance estimates are larger compared to when using an oracle. 

(iii) The second contribution is to point out errors in the open source implementation. The first source of error is that in the implementation of Q-Prop/IPG adaptive normalization is applied to only a few terms which can introduce a bias. Same sample reuse (as done in open source implementations) can lead to biased estimation of policy gradients. After removing the bias in the policy gradients, there is no advantage observed in state-action baselines.

I am not an expert in RL. But with my limited understanding,  I can see that the paper clarifies some important misconceptions. It would have been great, if the authors explained in detail why action/state baselines provide improvement in discrete problems.
review_title: State-action baselines do not help reduce of policy gradients when used for continuous control problems.

final_score: rating: 6: Marginally above acceptance threshold
review_content: In the context of policy gradient methods, the state-dependent baseline has been used to reduce the variance of the gradient estimate. More recently several papers have suggested the use of state-action-dependent baseline, in order to further reduce the variance. This paper empirically studies the benefit of such a state-action-dependent baseline for various control tasks.
Its main theoretical contribution is the decomposition of the variance to several components, corresponding to the variance coming from (roughly speaking) the randomness of trajectory, a baseline-related term, and the randomness of the advantage-weighted gradient of the log-probability of the policy over states.

The paper empirically shows that for certain benchmarks and a certain architectural choice for NN, the variance term corresponding to the baseline contributes less than the term related to the randomness of the trajectory. Therefore, it does not matter much to use a state-action-dependent vs. state-dependent baselines.

The paper tries to justify the success observed in previous work that used state-action-dependent baselines. It identifies some bugs in each implementation, which explains the apparent superior performance. In other words, the benefit was not due to the use of state-action-dependent baseline.

I believe the paper has an interesting observation, albeit with a quite limited scope. The paper does not show an impossibility result that state-action-dependent baseline cannot help (most likely it actually does for some problems, as mentioned in the Discussion section). It does not characterize when we can expect benefit from the action-dependent baseline and when we cannot. What it shows is that for some specific problems, using state-action-dependent baseline, implemented with a particular choice of architecture, does not help. I would consider this as something that might be good to be known by practitioners.

Also the fact that the authors have identified the bugs in other people’s implementations is commendable, and probably helps improving the available codebase.

So overall, I think this is an alright workshop paper, but its contributions are not major.
review_title: An interesting observation but with a limited scope

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The paper is well written, and the key contributions are supported by illustrative experiments. I reviewed the arXiv version of the paper (Feb 27 version); a few minor typos still remain (e.g. eqn in 9.2 needs a superscript \Sigma^a). Interesting paper that decomposes the sources of variance for RL policy gradients, identifies bugs in prior work, and proposes a simple modification that works.

Significance: Good. Isolates bugs in prior art, cleanly motivates fixes. The specific parametrization of horizon-aware value functions (Section 5) can be better motivated -- from a stylized variance formula for LQG to the \hat{V} eqn in Section 5 is a big leap! If the parametrization was argued for in a more principled way, I'm sure the paper becomes much more significant (can become the default way for parametrizing value functions).

Clarity: Good. There were a few expt details that were not clearly listed (e.g. for figure 2/Appendix 10, what was the policy pi? Is the x-axis measuring the same quantity as Fig 1 (task horizon)? Was pi being updated online as we move along the x-axis?)

Quality: Excellent.
review_title: Interesting paper that decomposes the sources of variance for RL policy gradients, identifies bugs in prior work, and proposes a simple modification that works

Meta-review: Congratulations, your paper was accepted to the ICLR workshop.
Decision: Accept