Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a general learning method for solving composite tasks in multi-agent RL settings. They propose to learn a high-level allocator with a low-level  action executor. To alleviate problems such as large action spaces, the authors introduce various techniques to make the learning possible. Finally, the authors applied ALMA to a firefighting problems and the SMAC benchmarks.
strengths_and_weaknesses: Strengths:
1. The paper is clearly written.
2. The problem is interesting. RL has shown great progress as a policy optimization tool. However, in many real-world applications, the structure of the tasks should be considered. 
3. The performance is better than baseline on various benchmarks. 

weaknesses:
1. The generality of ALMA is not convincing. For different tasks, the task allocator might have different action spaces. In some certain problems, the scheduling problem can be solved by other non-deep methods. The authors only empirically show  that the proposed method is better than others without sufficient discussion.
2. The masking trick seems important. But this is pure inductive bias.
3. The testing environment are on the simple end of the spectrum. It would be nice to have a robot task in this paper.
questions: Can you discuss more in depth about when your allocator is better than a method with strong heuristics?
 
limitations: We appreciate the author for being upfront. The limitations are 1) the exploration are simple e-greedy; 2) observation masks can be discovered automatically.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper presents a method for achieving hierarchical multi-agent learning. The proposal network is trained to assign subtasks to each agent and each agent will be assigned to accomplish its own task. The proposed method is evaluated on the save the city tasks and starcraft II multi-agent micro-management tasks. It shows better performance on the benchmarks. 
strengths_and_weaknesses: Strength: 
1. The paper presents an interesting idea of hierarchical multi-agent learning using the pointer network. The task assignment and skill learning in the multi-agent setting is a good approach and can be applied to many real-world tasks.

2. The writing is clear and the formulation is straightforward. 

Weakness: 
1. My main concern about the paper is lacking baselines. I think in order to show the superior performance of hierarchical multi-agent learning, we need to compare it against the simple centralized training and decentralized execution method. I didn't see the comparison in the Starcraft setting against papers like QMIX, RODE, ROMA. I think will be convincing if the paper compares the proposed algorithm with these methods, in order to show the benefit of hierarchical learning. 

2. I am a little confused about the performance in the Starcraft setting since the results in QMIX, RODE and ROMA are much better than the presented results in FIg. 4. I am wondering if there is some difference in the setting, or is it a fair comparison?

3. I am very interested to see some ablation studies, like the entropy coefficient in Eq. 3 and how the frequency of assigning subtasks to the agent will affect the performance. In particular, if the agent doesn't accomplish the assigned task when the high-level controller decides to change the task, what will happen?

4. It's a minor issue but I feel it will make the paper more convincing if we can test more tasks on the Starcraft settings.
questions: Please see the weakness above. 
limitations: I didn't see some potential negative social impact.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents a method of hierarchical multi-agent reinforcement learning, specifically for the case where there is a high-level centralized allocator that assigns low-level subtasks to different agents. It assumes that the environment POMDP has some predefined subtasks, from which we can learn low-level policies, and additionally learn the high-level allocator policy simultaneously. The main challenges include handling variable state and action spaces, as well as handling a very large action space for the allocator policy which are addressed via attention models (state) and pointer networks (action). The paper runs experiments on SaveTheCity and StarCraft, two challenging multi-agent environments that involve different number of agents in different configurations. Finally the paper speculates that this formulation may have applications in warehouse robotics.
strengths_and_weaknesses: **Strengths**

- The paper is well-executed, in the sense that it identifies practical challenges with the allocator-actor framework and addresses them in very sensible ways. For example, dealing with large action spaces using Amortized Q-Learning and proposal networks, using pointer networks and attention models.

- The experiments are done on challenging environments, and were tweaked to be even slightly more challenging (introducing more enemies in StarCraft), which is impressive. The results are convincing too.

- The writing is exceptionally clear. It gives enough clarification on background techniques and leaves enough room to describe ALMA in detail. Of particular note are the wonderful figures (Figs 2, 3), which are not just helpful/informative, but are truly some of the prettiest I've ever seen.

**Weaknesses**

My main reservation about this paper is the motivation of the problem formulation. There are many, many different setups in hierarchical MARL, each with different assumptions on the environment, observability, centralized/decentralized, etc.... While the Related Work section was helpful, I didn't feel that it was comprehensive enough to really convince me of the importance this allocator/actor framework. Doing a better job in this section would prevent this from feeling like 'yet another MARL framework'.
questions: L83: "We note that work in multi-agent task allocation often assumes the execution of tasks to be trivial"
L95: "however, they assume the pre-existence of low-level action policies"

Several points were made regarding the difference in ALMA being that it learns low-level action policies as well. But, it seems that we are assuming the POMDP itself has predefined subtasks with its own rewards. To me, this assumption makes learning the low-level action policies trivial, since we can just run any RL algorithm on the subtasks. Is my understanding correct?
limitations: .
review_title: null

Meta-review: metareview: The paper addresses multi-agent coordination in hierarchical task scenario by simultaneously learning coordination and execution policies. 

In summary, the reviewers found, and I concur, that the paper would be a welcome contribution to the NeurIPS community. The reviews found the paper clearly written, addressing an important problem, with good results. The rebuttal added new ablations and addressed the most critical reviewers comments. The final version should discuss methods the limitations.
confidence: Certain
Decision: Accept