Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper studies the problem of policy learning in a factored Markov decision process (MDP). The factorization of the transition probability distribution is graphically described through a set of influence diagrams. The objective function is a linear combination of weighted kernel functions. The authors propose an efficient policy gradient algorithm to learn the optimal policy. Simulation results support the authors’ results.
main_review: This paper considers an interesting problem in policy learning factored MDPs. I haven’t checked the details of the proof, but the proposed algorithm seems reasonable. Since many decision-making problems in the real world where its system dynamics can be naturally decomposed into a set of independent components, the method presented here could have many practical applications across disciplines. 

In simulations, the authors mainly compare against a vanilla policy gradient that does not exploit the underlying factored representation. I am curious how the authors’ method compares against other policy learning algorithms for factored MDPs, e.g., (Guestrin et al., 2003, 2004). Could the authors further elaborate on this?

limitations_and_societal_impact: - This work is mainly theoretical. Its long-term societal impact is unclear to see.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a policy gradient algorithm for MOMDPs with large factored action spaces, where the background knowledge on the MOMDP structure is given as an influence network. A new baseline for policy gradients is given that makes use of the MOMDP background knowledge and a factorisation of the RL policy. The paper analyses the favourable variance properties of the resulting policy gradient estimator. Empirical results are provided on a pedagogical bandit example, and a traffic control problem. The latter demonstrates applicability of the method even when the graphical assumptions only hold approximately. 
main_review: ### Originality
+ The paper is the first I've seen to use causal background knowledge in this way to reduce the variance of policy gradient methods. 
+ Graphical causal models have been used before in multi-agent problems (citation 27) but not for use in policy gradient methods on single-policy MOMDPs. 

### Quality
+ I enjoyed the results of section 4.1 and the discussion of a bias-variance tradeoff in practice. 
+ The experiments were thorough in the sense of studying properties of the approach beyond just performance. For example, the bias-variance tradeoff explored in figure 5b. 
- In figure 5a, it does not seem meaningful to plot standard error bars for 5 repeats. Perhaps the min and max across repeats would be more informative. In 5b, with only 5 repeats one could just plot all of the repeats instead of giving the standard error bar. 
- One of the main arguments for this approach in practice over using an action dependent baseline seems to be wall-clock time. Therefore I think including plots of this statistic in the appendix would be very valuable. 

### Clarity
+ In general the writing is easy to follow and the figures ease the exposition well. 
- Citation (56) computes the minimum variance baseline under the assumption that $\langle z_i, z_j \rangle =0$ (using the notation of the appendix). The paper nor appendix doesn't go far in explaining why in general settings we would not expect $\langle z_i, z_j \rangle =0$ to hold. 
- I think when introducing the action-dependant baseline in the experiments a citation should be put after it for clarity (eg (56)). 
- The paper describes the minimum factorisation approach as "principled" (in the main text) and "theoretically grounded" (in the appendix). It seems that this is done due to the existance and uniquenesss results. I therefore think the language used is slightly misleading, since it would imply to me that it is somehow the optimal factorisation (for, say, variance reduction) however there does not seem to be a result of this sort. Some discussion of the variance properties is given on line 263, but if there is a formal result regarding this, I think it should be written as such. 
- Line 284 refers to figure 7. However, this is a figure in the appendix. The same is done with figure 9 on line 295. 
- In line 322-324, it was clear to me what methods 1 and 3 were. However it was not clear what method 2 was. Was this the same as the action dependent baseline? 
- Why did the action dependent baseline take two order of magnitude more wall-clock time? This was stated in section 5, but not explained. Was it due to fitting $\omega$ (notation of appendix) each time to compute the baseline? 

### Significance
+ The method is simple, has nice theoretical results, and solid empirical performance. The use of background knowledge (influence graph) is reasonable and experiments show robustness to the assumptions not holding. Improving sample efficiency of policy gradients is certainly a direction many people are interested in. 
- Traffic control problem is the only non-pedagogical example used for experiments. Demonstrating more general applicability would be nice. Perhaps in other cases it is not so easy to write down an approximate influence graph. 
- The paper assumes that there is a scalarisable additive reward, which in MOMDPs is not true in general without prior knowledge. 
limitations_and_societal_impact: I was satisfied with the discussion on societal impact in the broader impact statement. I felt that the limitations were transparently presented in the paper.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper studies multi objective MDPs with a factorized action space and proposes a new “baseline” for policy gradient related actor-critic sorts of approaches for the class of problems that are of interest to them. They show variance reduction and perform experiments to gauge the power of the proposal. 
main_review: On the face of it, the paper makes contributions that seem quite reasonable to the actor-critic and policy gradient literature for a specific class of MDP problems: those involving factored action space and multiple objectives. I’m sure this class of problem is large enough such that the paper provides useful contributions to the literature. However, I do not have expertise in these subjects and will defer to my co-reviewers to gauge the work in this space. 

My main concern with the paper is that it seems to have a major positioning problem. I have more expertise in graphical models and causality – and the problem with the paper’s positioning is probably why the paper was assigned to me. Unfortunately, the paper has very little to do with causality, even though the authors choose to mention causality to describe their factored model. There seems to be a trend these days where people add the word “causal” to terms. I find the phrase “causal policy gradients” to be unsuitable for their approach for many reasons, but primarily as it seems to imply that it applies to a broader class of problems than it really does. I strongly recommend changing the term and re-positioning the paper. The fact that problems can be factorized as described in the paper and that values could depend on different decision (action) variables is not at all surprising, and there is no need to mention causality (I mention some relevant literature on graphical models later). If the authors disagree, they should feel free to explain their choice of terminology. 

Detailed comments:

P1: What are “daemon agents”?

P2: The word “baseline” is used in a few places on this page, but it is not until much later that it becomes clear what this refers to. I suggest the authors modify the introduction to mention actor-critic and other ideas earlier, so that the contributions make sense to a general AI reader.

P3: The description of factorization in the action space was not totally clear as described. What does it mean for actions to be “accessible” in an element-wise fashion? Does it simply mean that the action space is a cross-product of individual action spaces, and that they could individual affect different aspects of reward (like in Figure 1)? A real example would help here, and in other places. 

P3: On line 126, psi_1 is mentioned to be equivalent to q – should this be v?

P4: The authors are perhaps not familiar with influence diagrams – these are graphical models for decision problems, including MDPs (including multi-objective one). Their influence network can be represented as an influence diagram, with arcs from decision/action variables to value nodes. The idea of factorization using additive value functions is well known in that literature. Here is an older paper primarily about dynamic problems, but the ideas extend to multi-objective problems too: 
J. A. Tatman and R. D. Shachter (1990), Dynamic Programming and Influence Diagrams, IEEE Transactions on Systems, Man and Cybernetics 20(2):365-379.

P4: I’m surprised that actions are never allowed to depend on other actions in a policy. What is the reason for this simplification?

P4: The authors mention the market making problem in Section 3.1 as an application. Could they provide more detail? More motivation around the problem setting would help the paper greatly.

P4: It looks like there is a self-reference in Def. 3.3.

P4: “n” seems to be used both for the number of actions in the action space (earlier) but also for the cardinality of Sigma, which can probably be less than “n” in general. There is also an “m” mentioned earlier. Could the authors clarify please? 

P5: As mentioned earlier, I find “causal policy gradients” to be an absurd term for the method. I suggest changing the title of Section 4. Perhaps “factored policy gradients” or something like that.

P6: I may have missed something but how can one tell the second term in Equation (8) is asymmetric. Is this inferred from the beta?

P7: There are terms in Def. 4.2 that were not defined. Also – I found most of Section 4.2 hard to parse. Whatever I did understand, I did not find surprising. I was not able to fully appreciate the implications of this section. From what I could tell, the main contributions of the paper are around the variance results in Section 4.1.

P7 and 8: Again, I did not appreciate Section 5.1. Perhaps this is because the example was abstract and did not give me a sense of the implications. How many objectives are involved in this synthetic example?

P8: Example 7 should probably be Example 4a.

P8: Why is “Flow” outstanding?

P9: I find line 355 (“one of the most promising directions …”) an overstatement and not at all backed by the paper. Please remove or rephrase that line.
limitations_and_societal_impact: The authors should clarify that their approach works for a specific sub-class of problems. Other than that, I don't see issues.
review_title: null

Meta-review: This paper introduces a novel causal baseline to reduce the variance in policy gradient methods. All reviewers like the paper and I recommend acceptance. 
Decision: Accept (Poster)