Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a framework for multi-agent reinforcement learning with the goal of selecting heterogeneous behaviors and allocating them to agents so that an optimal policy can be achieved. The proposed framework represents skills as latent variables which are used to assign skills so that in the end agents can learn heterogeneous policies. The proposed framework is tested on three different multi-agent tasks and compared with other methods. In addition, an ablation study is included.
strengths_and_weaknesses: The paper appears to inspiration from a few other papers that are cited, including [4] and [32], but the view taken by the paper in identifying skills to improve MARL is significant to potentially achieve cooperative tasks. The paper is also overall well written, providing the intuition on the main choices made for the proposed framework and presenting sound technical details. It is appreciated also the experiments in three different problems and the evaluation with other methods, as well as the ablation study.

While the paper shows an interesting direction, the tasks are relatively constrained making the significance of the proposed framework more limited. Indeed, as noted also in the limitations, the framework does not appear to be able to learn appropriate policies when more skills are present.

In addition, the paper could discuss not only the literature on MARL, but also classic methods of heterogeneous multi-agent task allocation, which is currently missing, including
- Wu, H., Ghadami, A., Bayrak, A. E., Smereka, J. M., & Epureanu, B. I. (2021). Impact of Heterogeneity and Risk Aversion on Task Allocation in Multi-Agent Teams. IEEE Robotics and Automation Letters, 6(4), 7065-7072.
- Korsah, G. A., Stentz, A., & Dias, M. B. (2013). A comprehensive taxonomy for multi-robot task allocation. The International Journal of Robotics Research, 32(12), 1495-1512.
- Emam, Y., Mayya, S., Notomista, G., Bohannon, A., & Egerstedt, M. (2020, May). Adaptive task allocation for heterogeneous multi-robot teams with evolving and unknown robot capabilities. In 2020 IEEE International Conference on Robotics and Automation (ICRA) (pp. 7719-7725). IEEE.
- Schillinger, P., Bürger, M., & Dimarogonas, D. V. (2018). Simultaneous task allocation and planning for temporal logic goals in heterogeneous multi-robot systems. The international journal of robotics research, 37(7), 818-838.
questions: In addition to commenting to the two points raised in "Strengths and weaknesses" -- i.e., the non direct applicability to complex scenarios and the comparison with non-MARL classic methods:

A few details that could make the paper clearer:
- the methods compared appear to be from the same group; it is worth to include and/or discuss the choice for the comparison. For example, MAVEN could be used as comparison.
- the differences between the scenarios in each multi-agent task are not clear, as only some arbitrary naming is used. 
- it is worth to make examples of skills and actions as the way the paper discusses them they appear to be the same.

There are a few language problems, including:
- "attracts widely attention" -> "attracts wide attention"
- space after ; -- e.g., "are represented;(2)" -> "are represented; (2)"
- "dispose of" -> "consider" 
- "With the representation" -> "With this representation"
- "one-hot" -> "one-shot"
- "should explore differently and access different states of the environment." There is different repeated, and "explore differently" could be more specific.
- "described as follows Equation (10)." -> "described as follows."
- "comparing results" -> "comparative results"
- "Another notification" -> "Another note"
- "sparce" -> "sparse"
limitations: The appendix includes a section where the main limitations are discussed and providing some some potential venues of future work. It is worth to discuss how the framework could scale to more complex real-world scenarios.

The section about the impact discusses that overall the method itself should not have a direct negative outcome, however notes that in practical applications they could occur. Some examples and ways to address them could be discussed.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The authors suggest a skill-based multi-agent learning algorithm that generates diverse skills for finding heterogeneous behaviours. In the proposed mehtod, there are three mechanisms for capturing heterogeneous skills: skill representation, skil selector and skill-based policy learning. Experiment results show that the proposed method outperformed than other multi-agent methods on three multi-agent RL benchmarks.

strengths_and_weaknesses: Strenghts
   
  1. The proposed method (HSL) suggests two types of intrinsic rewards for skill selector and policy learning to enable effective exploration on both skill selection and heterogeneous behaviour.

  2. Experiment results show the significant improvements on several benchmarks.

Weakenesses
  1. The reviewer does not understand why the skill representation mechanism is needed in the proposed framework. 
     1) Two decoders can be trained without the latent skill varialbe, since two decodes already take observation and actions. So I could not agree that the auto-encoder generates latent skill variables that reveal different effect of different skills.
     2) In equation (2), the cosine distance is used to distinguish latent skill varialbes, but I think the skill-id is already enough for distinguishability. 
  
  2. The motivation of introducing $H[a|o,z]$ for intrinsic reward $r_m$ is not acceptable. The reviewer thinks that the effect of maximizing policy entorpy is just effective exploration not discrimability of skill.

  3. There is something wrong with constructing the lower bound of the intrinsic reward $r_m$. 

        In equation 4, two terms 1 and 2 are conditional entropy, but the author use these conditional entropy as just common shannon entropy.
        1) $H[a|z, o] =  \mathbb{E}_{p(z|o)}[H(p(a|z,o))] \neq H(p(a|z,o))$ is the conditional entropy between random variable $a$ and $z$ given $o$.
        2) $H[z|o] =\mathbb{E}_{p(o)}[H(p(z|o))] \neq H(p(z|o)) $  is the conditional entropy between random variable $z$ and $o$
questions:    1. Paragraphs on lines 58-70 should go to the relevant work section.

   2. There is no description how to compute the local $Q_i^v$ in Secion 3.2

   3. The loss function for $q_{\theta_z}$ is missing.

   4. As previously explained, the reviewer believes that the skill-id is sufficient to be used as a latent skill variable. Could you provide the results of using skill-ids as skill varaibles instead of skill representation method.
limitations: yes
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors proposed a skill-based MARL framework to enable agents to master diverse skills. The framework consists of the skill representation mechanism with an auto-encoder model, the skill selector to realize the assignment from agents to skills, and the skill-based policy learning mechanism with a mutual information-based intrinsic reward function. Experimental results show that the framework obtains the best performance on three challenging benchmarks, i.e., StarCraft II micromanagement tasks, Google Research Football and GoBigger, over state-of-the-art MARL methods.

strengths_and_weaknesses: The strength of the paper is as follows:
* Novelty: The authors proposed the skill representation mechanism with an auto-encoder model, the skill selector to realize the assignment from agents to skills, and the skill-based policy learning mechanism with a mutual information-based intrinsic reward function.  
* Clear results: Experimental results show that the framework obtains the best performance on three challenging benchmarks, i.e., StarCraft II micromanagement tasks, Google Research Football and GoBigger, over state-of-the-art MARL methods.

The weakness of the paper is as follows (please also see below “Questions):
* The presentation was sometimes inconsistent.
* The method descriptions were sometimes unclear. 

questions: * L45- “However, extra introduced networks for all agents hinder the application of CDS on large-scale tasks, which will be illustrated in the experiment.” The proposed HSL requires extra networks to obtain the skill diversity. Since HSL outperformed CDS in MARL performance, the problem of CDS can be mentioned differently. 
* L248- In association with the above, “Still, the reason is that the number of agents in this scenario is much higher than that in the other two scenarios, which leads to an adverse effect on training extra networks in CDS”. I can understand the former (“the number of agents in this scenario is much higher than that in the other two scenarios”), but the latter (“which leads to an adverse effect on training extra networks in CDS”) was unclear. What did the authors try to mention? 
* Figure 1: How did the authors create the Skill_ID pool? I cannot find this. In particular, I want to know whether this is created in a rule-based or data-driven manner. 
* L142- “This procedure is essentially a many-to-many assignment problem”. However, I cannot find how to solve this problem. How did the authors solve this problem? 
* Appendix A: Eq. (12) is different from eq. (4). If s can be replaced with o, but what is x? Where is the 3rd term of eq. (4)?
In Appendix Table 2, n_skill of Hard Counter-attack in GRF was 3, but in Appendix C.1 Skill Demonstration on GRF, the authors trained HSL with 4 skills. Which is correct? 

limitations: The limitations and potential negative societal impact were described in Appendices D and E, respectively. However, the latter was not concrete (“Therefore, we do not anticipate a direct negative outcome. In practical applications involving our method, potentially negative outcomes might occur.”).
review_title: null

Meta-review: metareview: The reviewers have largely agreed upon the value of the paper's concept (heterogeneous skills for MARL) and appreciated its impressive experimental gains on a range of environments. Each reviewer pointed out unique areas for improvement: citations to classical work, precision in derivations around conditional entropy and general writing improvements - which I find were largely addressed in the rebuttal discussions.

Although the precise notion of skills and their application in the MA setting can be debated (is it just an exploration guide, as stated by reviewer j8Gw?) that is quite an interesting debate to engage in. The relatively unique concept for the area and strong performance on a core set of benchmarks will be quite interesting for the NeurIPS community.  
confidence: Less certain
Decision: Accept