Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: Sharing parameters among all agents usually limits the behavioral diversity while learning a separate policy for each agent without any parameter sharing usually leads to high training complexity.
To balance the training complexity and the diversity of agents’ behaviors, this paper proposes a dynamic parameter sharing framework LDSA. LDSA divides the agents into up to $k$ groups (where $k$ is a manually set parameter) according to their action-observation histories. All agents share the parameters of the input layer and the GRU layer of the individual Q network. For the output layer, only the agents within the same group share the same parameters and the agents in different groups use different parameters. Experimental results in SMAC valid the benefits of the proposed dynamic parameter sharing mechanism.
strengths_and_weaknesses: **Strengths**
  * The paper is well motivated and clearly written.
  * The proposed framework LDSA is technically sound.

**Weaknesses**
  * (1) The proposed method has little to do with "dynamic task assignment". The core of LDSA is dividing the agents into up to $k$ groups based on their different behaviors and making the agents in different groups do not share parameters. There are no explicitly pre-defined tasks for the agents to solve. Therefore, using the words "dynamic parameter sharing according to agents' behaviors" may be more appropriate.
  * (2) Important related works and baselines are missing. [1] and [2] propose different methods to balance the training complexity and the diversity of agents’ behaviors by dynamically sharing parameters among agents. I notice that the authors cited these two works in the paper but not discussed and compared with them. These two works should be well discussed in the related work and compared in the experiments.
  * (3) The experimental evaluations in SMAC is not convincing. Recently, [3] and [4] have verified that the optimized QMIX (completely sharing the parameters among all agents) could achieve 100% win rates on all Easy, Hard and Super Hard scenarios of SMAC. Therefore, SMAC may not be a good testbed to validate the benefits of the dynamic parameter sharing mechanism. The authors could conduct more experiments in more complex environments, e.g., the challenging Google Research Football (GRF) [5].


**Reference**
  * [1] Christianos F, Papoudakis G, Rahman M A, et al. Scaling multi-agent reinforcement learning with selective parameter sharing[C], 
  * International Conference on Machine Learning. PMLR, 2021: 1989-1998. 
  * [2] Chenghao L, Wang T, Wu C, et al. Celebrating diversity in shared multi-agent reinforcement learning[J]. Advances in Neural Information Processing Systems, 2021, 34: 3991-4002.
  * [3] Hu J, Jiang S, Harding S A, et al. Rethinking the implementation tricks and monotonicity constraint in cooperative multi-agent reinforcement learning[J]. arXiv e-prints, 2021: arXiv: 2102.03479.
  * [4] Hao X, Wang W, Mao H, et al. API: Boosting Multi-Agent Reinforcement Learning via Agent-Permutation-Invariant Networks[J]. arXiv preprint arXiv:2203.05285, 2022.
  * [5] Kurach K, Raichuk A, Stańczyk P, et al. Google research football: A novel reinforcement learning environment[C], Proceedings of the AAAI Conference on Artificial Intelligence. 2020, 34(04): 4501-4510.
questions: Please see the weaknesses above.
limitations: The group number $k$ has to be manually set aforehand for each task.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper introduces an algorithm for efficiently training diverse abilities/policies in cooperative multi agent tasks. It learns to decompose a task into subtasks, match agents to a subtask based on their action-observation trajectory, and train policies for each subtask from the experience of agents assigned to it on that timestep. It does better than QMIX, ROMA and RODE on the harder tasks on the StarCraft II micromanagement tasks.
strengths_and_weaknesses: Strengths:
- Well organized and clearly written paper.
- Original and elegant method for decomposing subtasks without prior knowledge, and also without limiting the action space per subtask (like RODE) which is more flexible. 
- It does beat the other algorithms on harder tasks, which is a significant contribution.
- The ablations, and other investigations (of e.g. effect of number of subtasks) helpful for understanding the different parts of the algorithm.

Weaknesses:
- Not sure why comparing only to QMIX style mixing networks makes for a fair comparison; why would comparing to other types of approaches not be fair?
questions: - In the ablations section: Is there a way to decrease LDSA parameters and compare to the original-sized QMIX rather than increase QMIX parameters?

limitations: Limitations - they mentioned it may not improve things on simple tasks. I would also add that this is formulated only for a fully cooperative game at the moment.
The author didn't discuss potential negative social impacts, it would be good to mention even if the answer is inconclusive.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work addresses skill/ability specialisation for agents sharing a controller in cooperative MARL. LDSA, the proposed approach in this work, assumes agents' skill specialisation is required to solve subtasks constituting a particular task. LDSA subsequently learns to assign agents to subtasks by:
1) Creating identically sized representations of subtasks and agents' abilities.
2) Match agents and subtasks according to the similarity of their representations.
3) Learning a policy that agents need to follow for each subtask.

The process of learning representations of agents & subtasks, alongside the optimal policy for each subtask, is done end-to-end in combination with QMIX, which is a value-based value factorisation MARL method. Aside from the temporal difference loss functions associated with QMIX, LDSA's optimised function also includes regularisation terms that (a) force subtasks to have different representations from each other and (ii) agents to have similar subtask assignment distributions between consecutive timesteps.

This work subsequently evaluates LDSA in various scenarios in SMAC. LDSA's performance is compared against QMIX and previous MARL methods for role discovery, such as ROMA and RODE. This work also conducted an ablation study to find the importance of the QMIX-based objective function alongside other regularisation terms for the trained agents' returns. These experiments' results empirically demonstrate LDSA's significantly improved performance compared to previous role assignment methods. Further analysis of the resulting agents' behaviour in the SMAC corridor scenario indicates agents' ability to solve diverse tasks to perform well in the environment.
strengths_and_weaknesses: **Originality**

**(Minor Weakness) LDSA is an incremental idea relative to the most recent works in the area.**

The idea of assigning agents to roles based on their observed trajectories has been explored before in the ROMA approach (Wang et al., 2020). While ROMA proposes an implicit role assignment method that conditions agents' policies on task representations, explicit approaches like LDSA that learn K separate policies for different subtasks and the optimal mixture of these policies has recently been explored by Nguyen et al. (2022). Compared to the idea from Nguyen et al. (2022), LDSA merely offers a different way of computing the weights and utilisation of policies of each subtask.

**Quality**

**(Major Strength) Comparison against several prior methods in role assignments show LDSA's significantly higher returns.**

By outperforming ROMA and RODE, this work demonstrates that LDSA has the potential of improving several existing approaches for role assignment. 

**(Major Strength) Empirical evaluation of learned subtasks.**

Empirical evaluation of learned subtasks in SMAC's corridor setup indicates that LDSA manages to learn meaningful subtask assignment in some environments. This experiment further illustrates the significance of this work's results and should ideally be done for all evaluated environments.

**(Major Weakness) Missing baselines on skill/ability specialisation for parameter sharing in MARL.**

Since the abstract and introduction section frames LDSA as a solution for skill/ability specialisation under parameter sharing in MARL, a comparison against CDS (Li et al., 2021) is necessary since it also attempts to achieve the same goal as LDSA. 

Beyond the similarity in their underlying goals, comparing LDSA against CDS may also help investigate the effects of one of LDSA's potential weaknesses. In particular, LDSA assigns a role to agents based on their observed trajectories. This can be a drawback when agents exhibiting similar trajectories must be assigned different subtasks to improve overall performance. For instance, consider a kitchen setup where two agents must clean dirty plates, chop different types of ingredients, and serve the chopped ingredients on the cleaned plates. Under this setup, the optimal policy is for both agents to jointly wash the dirty dishes, divide the chopping of different ingredients among them, and subsequently serve the ingredient they have chopped on a plate. Note that before ingredients are chopped, the trajectory of both agents is similar since they are solving the same task up to that point. This lack of a symmetry-breaking mechanism to assign agents with similar trajectories to different subtasks is not present in CDS since each agent in CDS has individual components not shared with others. 

**(Major Weakness) Missing baseline to highlight the importance of the proposed role assignment method.**

Since LDSA main novelty is its proposed role assignment method, the baselines can be useful to highlight the importance of LDSA's role/subtask assignment components:

1. A baseline where task assignment probability is directly estimated from agents' observed trajectories. Comparing LDSA's performance against this baseline helps us understand the importance of mapping subtasks & agent trajectories to fixed-length representations and computing their similarities to compute task assignment probability. 
2. A baseline where an agent employs a mixture of the individual subtasks' policies rather than using a single sampled policy.

Note that these two aspects are the main differences between LDSA and the approach proposed by Nguyen et al. (2022). Thus, comparing LDSA's performance against these may provide important insights regarding which role assignment approach is optimal.

**(Minor Weakness) Lack of diversity in environments used for evaluation.**

While this only provides a minor improvement, the generality of this work's results can be further demonstrated by showing it works in a wider range of environments. For instance, Google Research Football and Overcooked are two environments where subtask assignment is required for learning.

**Clarity:**

**(Major Strength) The work is generally well written and very easy to understand.**

Other minor suggestions to improve clarity will be provided in the questions/suggestions section below.

**Significance:**

**(Major Strength) The results of this work has potential significance to people focusing on dynamic task assignment in MARL.**
 
The idea of jointly learning subtask and ability representations can be an essential contribution if it is demonstrated to perform better compared to the baselines mentioned above. However, further experiments are required to demonstrate the significance of this work.

**Citations:**

Wang et al. 2020. ROMA: Multi-Agent Reinforcement Learning with Emergent Roles. ICML 2020

Wang et al. 2021. RODE: learning roles to decompose multi−agent tasks. ICLR 2021.

Nguyen et al. 2022. Learning to Transfer Role Assignment Across Team Sizes. AAMAS 2022.

Li et al. 2021. Celebrating Diversity in Shared Multi-Agent Reinforcement Learning. NeurIPS 2021.

questions: **Questions**

1. How does LDSA deal with the need for assigning different roles to agents with similar observed trajectories?
2. How will LDSA perform compared to a method that directly estimates the mixture weights of each subtask policy as opposed to computing it based on agent ability-subtask representational similarity.
3. What happens if LDSA uses a mixture of subtask policies (that are weighted according to inferred component weights) as opposed to sampling one subtask policy at any timestep?
4. In Figure 6 on Section 4.4, what is the difference between the subtask denoted by the red and blue lines?
5. In the problem formulation, why does subtasks have separate transition function compared to the solved task? Isn't it the case that tasks and subtasks are situated in the same environment (and hence sharing the same transition function)?

**Minor suggestions for writing**

1. Consider using different names for the two trajectory encoders in Lines 160 and 185.
2. For each environment used in evaluation, consider mentioning the different roles that are required for solving each environment. 
limitations: As indicated by the feedback provided in the previous sections, the authors have not highlighted the potential limitations of their work. Furthermore, I believe no statement regarding potential negative social impact of the work is required since this work is currently a generic MARL method that has yet been applied to real-world problems.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes an algorithm to learn dynamic subtask assignment (LDSA) in cooperative MARL. First, it uses a subtask encoder to construct a vector representation for each subtask. Then, it proposes an ability-based subtask selection strategy to dynamically group agents into the similar subtask. In addition, the authors employ two tricks to make the representations of subtasks distinguishable and the learning process steady. Empirical results show that LDSA learns reasonable and effective subtask assignment for better collaboration and significantly improves the learning performance on the challenging StarCraft II micromanagement benchmark. 
strengths_and_weaknesses: Strengths: 
- The writing of the paper is clear and structured. 
- Previous works dealing with multi-agent cooperative tasks predefine the task decomposition using prior domain knowledge, while this work does not use prior knowledge to decompose the task. 
- The number of subtasks should be set in advance but is a robust hyperparameter that all the scenarios in the SMAC benchmark can share an identical number of subtasks. 
- The ablation study demonstrates the effectiveness of the two regularizations and different components of LDSA. 

Weaknesses: 
- Since the number of subtasks $k$ can vary, the subtask defined in Definition 1 is more self-defined than objectively existing, making the definition informal. We think the authors should clarify the assumption under which the overall task can be decomposed into subtasks. 
- Ground truth about the subtasks is not available, so we prefer that this method works on the SMAC benchmark because it encourages the diversity of agents’ policies which encourages exploration. We think it would be better if the authors could clarify the assumption that the more diverse the policies, the better the agents would perform. 
- The experiments are only carried out on the SMAC benchmark, which restricts the reliability of the results. As far as we know, after fine-tuning, even a baseline like QMIX can achieve excellent performance on SMAC (GitHub - hijkzzz/pymarl2: Fine-tuned MARL algorithms on SMAC (100% win rates on most scenarios)). We believe it would make the results more convincing if the authors could conduct more experiments on other types of environments. 

questions: - We wonder how this method would perform if the multi-agent cooperative tasks do not require diverse agents' policies. For example, in Level-Based Foraging (GitHub - semitable/lb-foraging: Level-based Foraging (LBF): A multi-agent environment for RL), agents need to cooperate in strong coordination to collect foods with different levels where different types of policies are not necessary. We wonder if the effectiveness of LDSA depends on the type of problem. 
- We wonder why the performances of ROMA and RODE are considerably worse than those in their original paper. For example, RODE can achieve above 80% winning rate on the map of corridor and 6h\_vs\_8z, while in this paper, RODE cannot learn anything after 5 million timesteps. 
- Are the trajectory encoders shown in Figure 1(b) & 1(c) the same? 
- As the authors claim in the main text, the agents learn different abilities after training, so we wonder how these agents could perform tasks different from the training phase, which are known as ad-hoc tasks. 

limitations: The authors do not claim the limitations of their method in the paper. We think, as mentioned above that the effectiveness of LDSA depends on the type of problem it aims to solve since the diversity of policies is not directly related to the effectiveness of cooperation. The authors should carry out more experiments on other different environments.
review_title: null

Meta-review: metareview: This paper proposes a method to dynamically group agents with similar representations and assign subtasks to each group so that they can effectively share parameters among agents within the same group while specializing across groups. The results on StarCraft Micromanagement benchmark and Google Research Football domain show that the proposed method outperforms relevant baselines including QMIX, ROMA, and RODE.

The reviewers found that the idea is interesting and technically sound, and the paper is very well-written. Although there were several concerns about the lack of baselines (CDC) and the lack of challenging benchmarks, the authors addressed most of them during the rebuttal period by updating the results with additional baselines, an additional benchmark (Google Research Football), and additional ablation studies. As a result, all of the reviewers agreed that the result is significant enough to be presented at NeurIPS. Thus, I recommend accepting this paper.
confidence: Certain
Decision: Accept