Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: 1. Summary

The authors apply MARL to principal-agent / mechanism design problems where selfish agents need to be incentivized to coordinate towards a leader's (collective) goal.

The leader is modeled as a semi-MDP with event-based policy gradients and modules to model/predict followers' actions. The leader sends messages to followers, an "event" is a pair (timestep, message of leader to a follower).
A `termination' menas that an agent should stop executing the previous selected action; the leader signals as such to the agent.
With this modeling step, the authors formulate an event-based policy gradient, which considers models for which goal to send to followers and when.

The authors compare this approach on 4 environments with M3RL, which also solves (extensions of) principal-agent problems.

2. Decision (accept or reject) with one or two key reasons for this choice.

Weak accept.

3. Supporting arguments

The approach seems sound and conceptually related to a multi-agent generalization of STRAW https://arxiv.org/pdf/1606.04695.pdf, where a planner predicts / commits to an action-plan for a single agent.

4. Additional feedback with the aim to improve the paper. Make it clear that these points are here to help, and not necessarily part of your decision assessment.

5. Questions
review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: This paper aims to improve the performance of leader agents in leader-follower Stackelberg Markov games (SMG) by introducing architectural components. an event-aware policy gradient for the temporally abstracted level of the leader, and a follower abstraction technique that mitigates the impact of follower distributional shift. The authors show significant improvements over the state of the art on several problems, and also evaluate the importance of the different components of their approach with thorough ablations.

I am not particularly familiar with the literature on SMGs, but I recommend this paper for acceptance. The architectural components (follower-wise attention), dense policy gradient, and follower-level abstractions are well-motivated, and the evaluations are extremely thorough. The supplementary material is longer than the main paper, and almost half of it is additional results.

The paper could be improved with more discussion of the applications of SMGs, although this could be due to my relative unfamiliarity with the area. For example, this seems immediately relevant to hierarchical RL problems that involve learning to coordinate multiple skills where the optimal subpolicies are incompatible, e.g. multi-robot coordination (or multi-limbed single robot coordination) with collisions. More discussion of the concrete applications would be informative.

Although I am not intimately familiar with the research area of SMGs, I expect this work will have significant impact in the area on the basis of the thoroughness of the results alone.
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper proposes an interesting event-based policy gradient algorithm for single-leader multi-follower Stackelberg Markov Games (SMG). The authors claimed 3 contributions on their work: model the leader's decision-making process as a semi-MDP and proposed an event-based policy gradient algorithm for the leader to consider long-term effect; exploit a leader-follower consistency scheme with a follower-aware module and a follower-specific attention module to better predict the follower's behaviors; and propose an action-abstraction policy gradient algorithm in order to make SMG easier to converge.

The proposed model is interesting and the authors have done sufficient empirical studies to show the success of their method, as well as some necessary theoretical derivations. My major concern about this work is whether the task that this paper follows is widely studied in the reinforcement learning community, as the authors mentioned that only [1] can be used as baseline methods since the other methods can not be applied in their problem. But in general I think this is an interesting paper. Some detailed comments and suggestions:

1. This paper proposed 3 major contributions: event-based policy gradient, leader-follower consistency scheme and action abstraction policy gradient. The authors show the empirical performance comparison for the case with and without each of these techniques in the experiments. It can be seen that all of the 3 techniques help the proposed method perform better. In addition, the proposed method performs better compared to the state-of-the-art method (in [1]). Hence the authors have shown the success of their algorithm empirically. One thing that can be done to improve this paper is to show a joint comparison for the 2 ^ 3 = 8 cases of with / without any of the 3 techniques so that we can gain a better understanding about the effect of these 3 techniques.

2. The authors provide some necessary theoretical derivations to compute the policy gradients. But there are some typos in the equations. For example, the numerator of the equation of the case e_i^k\not\in A_T in I(e_i^k) at the bottom of page 4 is incorrect. In addition, the equation on the second last line of page 3 should be P_\gamma(s_{t+1}, \omega_t, a_t | s_t, \omega_{t-1}), not P_\gamma(s_{t+1}, \omega_t | s_t, \omega_{t-1}, a_t).  Please check the details and solve these typos.

3. It will be better if the authors can provide more theoretical analysis or quantitative explanation about the proposed method. For example, why a dense EBPG better than a sparse one? Why the equations in the attention mechanism appear in their form, not some similar forms (e.g. scale the values from the function A before putting them into the softmax function)? Intuitively the proposed techniques make sense, but it will be great if more insights are provided.

4. As I mentioned, I am concerned if the proposed problem is widely studied in the reinforcement learning community. Also, since the authors mentioned in the appendix that two of the tasks are original from [2], but we can not apply the method in [2] as a baseline method. It is possible to modify the method in [2] to fit with the setting and compare with them?

5. It will be better if the authors can rearrange the paper for a little bit, since there are too many important details (e.g. experiment settings, assumptions in the theorems) are in appendix.

Questions:

1. Can the authors try to modify the method in [2] and compare with it (details in my comment 4)?

Reference:

[1] Tianmin Shu and Yuandong Tian. M3RL: Mind-aware multi-agent management reinforcement learning. In ICLR, 2019.

[2] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, OpenAI Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-competitive environments. In NeurIPS, pp. 6379–6390, 2017.
review_title: Official Blind Review #3

Meta-review: This paper tackles the challenge of incentivising selfish agents towards a collaborative goal. In doing so, the authors propose several new modules. 

The reviewers commented on experiments being extremely thorough. One reviewer commented on a lack of ablation study of the 3 contributions, which was promptly provided by the authors. The proposed method is also supported by theoretical derivations. The contributions appear to be quite novel, significantly improving performance of the studied SMGs.

One reviewer mentioned the clarity being compromised by too much material being in the appendix, which has been addressed by the authors moving some main pieces of content to the main text. 

Two reviewer commented on the relevance being lower because of the problem not being widely studied in RL. I would disagree with the reviewers on this aspect, it is great to have new problem brought to light and have fresh and novel results, rather than having yet another paper work on Atari. I also think that the authors in their rebuttal made the practical relevance of their problem setting sufficiently clear with several practical examples. 
Decision: Accept (Poster)