Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper describes a role-based learning model for the DEC-POMDPs. The main contribution lies in the efficient discovery of roles from the joint action spaces and then learning a bi-level role assignment for each achievement. This is achieved in two steps. First, the joint action space is clustered into different role action spaces that reduce the action search space for each role. Second, a bi-level role assignment technique is used to learn action and roles for each agent. The technique is tested on StarCraft II micromanagement environments.

For the action space reduction, the model learns action representations that can reflect the effects of actions on the environment and other agents. To this end, a deep learning model is created which predicts the effects of joint actions on the induced rewards and change in the effects. Actions generating similar effects are cluster together using K-means and are called roles action spaces. This restricts the joint action search spaces for each role. The role selector is now used to learn a bi-level hierarchical assignment to map the action-observation history of each agent. At the top-level the agents are mapped to their corresponding to roles based on a Q-value function of each role conditioned on action-observation history and at a lower-level similar Q-value function is used to find the agent’s action. To avoid too many concurrent selections of a single role and action by multiple agents, a global Q-value is learned from individual Q-values to ensure overall coordination between the agents. This is inspired by QMIX, previous work on multi-agent learning.

Positives:
1. The idea of reducing the search space by effect-based clustering appears interesting and novel.
2. The technique leads to good exploration and performance in hard and super maps.
3. The paper is well-written, and the technique is extensively tested on all the maps with useful ablations

Minor issues:
1. Some comments/reasoning related to outlier roles and action spaces would have been helpful
2. Do changes in the clustering algorithm leads to a significant difference in performance or role assignment?
3. Compared to the previous approaches, the RODE-algorithm learns slower in most of the easier maps. 

Update - Thank you for the response and updates to the paper
review_title: A paper with a clear contribution

final_score: rating: 7: Good paper, accept
review_content: This paper introduces a bi-level hierarchical framework for achieving scalable multi-agent learning. In this framework, the high-level policy (role selector) coordinates role assignments in a smaller role space and at a lower temporal resolution. And the low-level policies (role policies) explore strategies in reduced primitive action-observation spaces. In this way, the complex multi-agent problem is decomposed into multiple sub-problems, which is easy to learn. The authors conduct the experiments in the StarCraft II micromanagement benchmark, compared with the state-of-the-art MARL methods.

Strength:
- The paper is well-written and easy-to-follow. The authors show the demo videos for a better demonstrating of the learned behavior.
- The introduced idea is original and interesting. I can see that the bi-level multi-agent coordination framework is of great potential in solving various multi-agent tasks.
- It is nice to demonstrate the generalization of the learned policies on unseen maps.
- Provide the code in the Supplementary Material.

Questions:
- The action clustering. How to choose the roles number k for clustering? According to prior knowledge? Can you explain the effect of different k numbers on the performance? 
- Is the policies for different levels trained simultaneously?

review_title: Official Blind Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes a new method called RODE, to learn roles for multiagent systems to improve the learning efficiency of MARL. Instead of exploring in the full joint action space, RODE first decomposes the action space into K clusters based on the different influences of each action on the environment and other agents. Then RODE trains a role selector and role policies conditioned on each agent’s action-observation history. Experiments show its superior performance compared with SOTA MARL algorithms.

********
This paper is highly related to the subject of ICLR and is well written. But I have some specific confusions listed below, waiting for the authors' reply.

The authors mentioned that RODE differs from previous role-based methods that require prior knowledge to learn roles. However, the action space decomposition also requires human knowledge, e.g., would a wrong number of k clusters hinder the final performance?

Given the local observation and all agent’s actions, RODE first learns the action representations by minimizing two prediction errors: the next local observation, and team reward. Then the role representation is calculated by averaging representations of actions in each cluster. What I am concerned is that whether this average representation can well represent the roles, which may further impact the estimation of role policies, since the action representations are used to generate the q-values of both role selector and role policies. 


Since in SMAC, the attack action must be given an enemy index, so the number of actions increases with the increase of the number of enemies. Transfer of RODE needs to manually add new representations to each cluster and then re-calculate the role representations, In this way, the role representations change. Does this influence the decisions of role selector and role policies? Furthermore, these two components cannot be trained on new maps, because the input and output of the mixing networks are different with a different number of agents. 

Some recent MARL algorithms to improve QMIX, such as AI-QMIX[1], Weighted-QMIX[2] should be discussed or compared.

I agree with the authors that the joint action space can be decomposed or classified based on the action effect or action semantics. ASN [2] firstly investigates the influence of actions on other agents, which they call the action semantics. I think these two works are inspired by the same property in MASs to improve multiagent coordination. Moreover, these two works both use SMAC as benchmarks. In this way, I’d like to see the comparison of ASN and RODE to further validate its superior performance.

[1] AI-QMIX: Attention and Imagination for Dynamic Multi-Agent Reinforcement Learning. arXiv preprint arXiv:2006.04222.

[2] Weighted QMIX: Expanding Monotonic Value Function Factorisation. NeurIPS 2020.

[3] Action Semantics Network: Considering the Effects of Actions in Multiagent Systems. ICLR. 2020.
review_title: Review

Meta-review: The paper proposes a two-level hierarchical algorithm for efficient and scalable multi-agent learning where the high-level policy decides a reduced space for low-level to explore in. All the reviewers liked the premise and the experimental evaluation. Reviewers had some clarification questions which were answered in the authors' rebuttal. After discussing the rebuttal, AC as well as reviewers believe that the paper provides insights that will be useful for the multi-agent learning community and recommend acceptance.
Decision: Accept (Poster)