Venue: ICLR 2018 conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: In this paper, the authors investigate variance reduction techniques for agents with multi-dimensional policy outputs, in particular when they are conditionally independent ('factored'). With the increasing focus on applying RL methods to continuous control problems and RTS type games, this is an important problem and this technique seems like an important addition to the RL toolbox. The paper is well written, the method is easy to implement, and the algorithm seems to have clear positive impact on the presented experiments.

- The derivations in pages 4-6 are somewhat disconnected from the rest of the paper: the optimal baseline derivation is very standard (even if adapted to the slightly different situation situated here), and for reasons highlighted by the authors in this paper, they are not often used; the 'marginalized' baseline is more common, and indeed, the authors adopt this one as well. In light of this (and of the paper being quite a bit over the page limit)- is this material (4.2->4.4) mostly not better suited for the appendix? Same for section 4.6 (which I believe is not used in the experiments).

- The experimental section is very strong; regarding the partial observability experiments, assuming actions are here factored as well, I could see four baselines 
(two choices for whether the baseline has access to the goal location or not, and two choices for whether the baseline has access to the vector $a_{-i}$). It's not clear which two baselines are depicted in 5b - is it possible to disentangle the effect of providing $a_{-i}$ and the location of the hole to the baseline?

(side note: it is an interesting idea to include information not available to the agent as input to the baseline though it does feel a bit 'iffy' ; the agent requires information to train, but is not provided the information to act.  Out of curiosity, is it intended as an experiment to verify the need for better baselines? Or as a 'fair' training procedure?)

- Minor: in equation 2- is the correct exponent not t'?  Also since $\rho_\pi$ is define with a scaling $(1-\gamma)$ (to make it an actual distribution), I believe the definition of $\eta$ should also be multiplied by $(1-\gamma)$ (as well as equation 2).
review_title: Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes a variance reduction technique for policy gradient methods. The proposed approach justifies the utilization of action-dependent baselines, and quantifies the gains achieved by it over more general state-dependent or static baselines.


The writing and organization of the paper is very well done. It is easy to follow, and succinct while being comprehensive. The baseline definition is well-motivated, and the benefits offered by it are quantified intuitively. There is only one mostly minor issues with the algorithm development and the experiments need to be more polished. 

For the algorithm development, there is an relatively strong assumption that z_i^T z_j = 0. This assumption is not completely unrealistic (for example, it is satisfied if completely separate parts of a feature vector are used for actions). However, it should be highlighted as an assumption, and it should be explicitly stated as z_i^T z_j = 0 rather than z_i^T z_j approx 0. Further, because it is relatively strong of an assumption, it should be discussed more thoroughly, with some explicit examples of when it is satisfied.

Otherwise, the idea is simple and yet effective, which is exactly what we would like for our algorithms. The paper would be a much stronger contribution, if the experiments could be improved. 
- More details regarding the experiments are desirable - how many runs were done, the initialization of the policy network and action-value function, the deep architecture used etc.
- The experiment in Figure 3 seems to reinforce the influence of \lambda as concluded by the Schulman et. al. paper. While that is interesting, it seems unnecessary/non-relevant here, unless performance with action-dependent baselines with each value of \lambda is contrasted to the state-dependent baseline. What was the goal here?
- In general, the graphs are difficult to read; fonts should be improved and the graphs polished. 
- The multi-agent task needs to be explained better - specifically how is the information from the other agent incorporated in an agent's baseline?
- It'd be great if Plot (a) and (b) in Figure 5 are swapped.

Overall I think the idea proposed in the paper is beneficial. Better discussing the strong theoretical assumption should be incorporated. Adding the listed suggestions to the experiments section would really help highlight the advantage of the proposed baseline in a more clear manner. Particularly with some clarity on the experiments, I would be willing to increase the score. 

Minor comments:
1. In Equation (28) how is the optimal-state dependent baseline obtained? This should be explicitly shown, at least in the appendix. 
2. The listed site for videos and additional results is not active.
3. Some typos
- Section 2 - 1st para - last line: "These methods are therefore usually more sample efficient, but can be less stable than critic-based methods.".
- Section 4.1 - Equation (7) - missing subscript i for b(s_t,a_t^{-i}) 
- Section 4.2 - \hat{Q} is just Q in many places
review_title: Useful idea for variance reduction with some issues in the experiments

final_score: rating: 7: Good paper, accept
review_content: This paper presents methods to reduce the variance of policy gradient using an action dependent baseline. Such action dependent baseline can be used in settings where the action can be decomposed into factors that are conditionally dependent given the state. The paper:
(1) shows that using separate baselines for actions, each of which can depend on the state and other actions is bias-free
(2) derive the optimal action-dependent baseline, showing that it does not degenerate into state-only dependent baseline, i.e. there is potentially room for improvement over state-only baselines.
(3) suggests using marginalized action-value (Q) function as a practical baseline, generalizing the use of value function in state-only baseline case.
(4) suggests using MC marginalization and also using the "average" action to improve computational feasibility
(5) combines the method with GAE techniques to further improve convergence by trading off bias and variance

The suggested methods are empirically evaluated on a number of settings. Overall action-dependent baseline outperform state-only versions. Using a single average action marginalization is on par with MC sampling, which the authors attribute to the low quality of the Q estimate. Combining GAE shows that a hint of bias can be traded off with further variance reduction to further improve the performance.

I find the paper interesting and practical to the application of policy gradient in high dimensional action spaces with some level of conditional independence present in the action space. In light of such results, one might change the policy space to enforce such structure.

Notes:
- Elaborate further on the assumption made in Eqn 9. Does it mean that the actions factors cannot share (too many) parameters in the policy construction, or that shared parameters can only be applied to the state?
- Eqn 11 should use \simeq
- How can the notion of average be extended to handle multi-modal distributions, or categorical or structural actions? Consider expanding on that in section 4.5.
- The discussion on the DAG graphical model is lacking experimental analysis (where separate baselines models are needed). How would you train such baselines?
- Figure 4 is impossible to read in print. The fonts are too small for the numbers and the legends.

review_title: This paper presents methods to reduce the variance of policy gradient using action dependent baselines when actions have conditionally independent factors.

Meta-review: The reviewers are satisfied that this paper makes a good contribution to policy gradient methods.
Decision: Accept (Oral)