LDSA: Learning Dynamic Subtask Assignment in Cooperative Multi-Agent Reinforcement Learning
Authors: Mingyu Yang
Jian Zhao
Xunhan Hu
Wengang Zhou
Jiangcheng Zhu
Houqiang Li
Venue: NeurIPS 2022 Accept (2022)
Keywords: Multi-agent Reinforcement Learning
Task Decomposition
Subtask Representation
Subtask Assignment

Abstract: Cooperative multi-agent reinforcement learning (MARL) has made prominent progress in recent years. For training efficiency and scalability, most of the MARL algorithms make all agents share the same policy or value network. However, in many complex multi-agent tasks, different agents are expected to possess specific abilities to handle different subtasks. In those scenarios, sharing parameters indiscriminately may lead to similar behavior across all agents, which will limit the exploration efficiency and degrade the final performance. To balance the training complexity and the diversity of agent behavior, we propose a novel framework to learn dynamic subtask assignment (LDSA) in cooperative MARL. Specifically, we first introduce a subtask encoder to construct a vector representation for each subtask according to its identity. To reasonably assign agents to different subtasks, we propose an ability-based subtask selection strategy, which can dynamically group agents with similar abilities into the same subtask. In this way, agents dealing with the same subtask share their learning of specific abilities and different subtasks correspond to different specific abilities. We further introduce two regularizers to increase the representation difference between subtasks and stabilize the training by discouraging agents from frequently changing subtasks, respectively. Empirical results show that LDSA learns reasonable and effective subtask assignment for better collaboration and significantly improves the learning performance on the challenging StarCraft II micromanagement benchmark and Google Research Football.

LDSA: Learning Dynamic Subtask Assignment in Cooperative Multi-Agent Reinforeement Learning
Mingyu Yang', JJan
Zhao', Jian
Xunhan Hu', Wengang Zhou Jiangcheng Zhu?, Houqiang Li
University of Science and Technology of China, "Huawei Cloud
3 Hefei Comprehensive National Science Center, Institute of Artificial Intelligence (ymy, , zj 140, cattyyyhhh
ustc. edu.cn zhyjiangcheng@huavei com, {zhug, lihg)oustc.......... www.t.eeu..c.........ooomm
Abstract
Cooperative multi-agent reinforcement learning (MARL) has made prominent progress recent years. For training efficiency and scalability, most of the MARL algorithms make all agents share the same policy value network. However, in many complex multi-agent tasks, different agents are expected possess specific abilities to handle different subtasks. In those scenarios, sharing parameters indis- criminately may lead to similar behavior across all agents, which will limit the exploration efficiency and degrade the final performance. To balance the training complexity and the diversity of agent behavior, we propose a novel framework to learn dynamic subtask assignment (LDSA) in cooperative MARL. Specifically, we first introduce a subtask encoder to construct a vector representation for each subtask according to its identity. To reasonably assign agents to different subtasks, we propose an ability- based subtask selection strategy, which can dynamically group agents with similar abilities into the same subtask. In this way, agents dealing with the same subtask share their learning of specific abilities and different subtasks correspond different specifie abilities. We further introduce two regularizers increase the representation difference between subtasks and stabilize the training by discouraging agents from frequently changing subtasks, respectively. Empir- ical results show that LDSA learns reasonable and effective subtask assignment for better collaboration and significantly improves the learning performance on the challenging StarCraft II micromanagement benchmark and Google Research Football.
Introduction
Cooperative multi-agent reinforcement learning (MARL) has recently received much attention due to its broad prospects on many real-world challenging problems, such as traffic light control [.) autonomous cars [2] and robot swarm control [3]. Compared single-agent scenarios, multi-agent tasks pose more challenges. On the one hand, the observation transition function of each agent is related the policies of other agents, which are constantly updated during training. Hence, from the perspective of any individual agent. the environment is extremely non- non-stationary. which will be exacerbated with more agents. On the other hand, the joint cectionobbeervation space of the multi- agent task grows exponentially with the number of agents. These two issues prevent MARL algorithms from scaling more agents.
To cope with a large number of agents, most of the recent MARL works, including value-based 4 and policy gradient [50 utilize technique of policy decentralization with shared parameters 16
*Corresponding authors: Wengang Zhou and Houqiang Li
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

whereby all agents share decentralized policy or value network. There are several merits that make parameter sharing so popular MARL. First and foremost, it considerably reduces the total number of trainable parameters and makes the learning complexity tractable. Besides, agents with the same parameters can share training experience with each other, which can effectively mitigate the slowdown of convergence rate due to non-stationarity [17]. However, many complex multi-agent tasks consist of a set of subtasks, where the transition or reward functions are distinct [8] [9]. Solving every subtask requires some specific abilities. In contrast, fully-shared parameters may cause all agents to behave similarly and hinder the diversity of agents' policies. For example, on the Google Research Football task (20), all agents will compete for the ball if sharing parameters [16] The similar behaviors across all agents will limit the exploration efficiency and degrade the final performance [2T]. Therefore, it's difficult for single neural network to learn the specific abilities required by all subtasks.
Alternatively, another solution to learn a separate policy for each agent without any parameter sharing, which allows diverse policies but leads to high training complexity. To balance the training complexity and the diversity of agents' behaviors, better method is to learn a distinct neural network for each subtask and group the agents by subtasks. Agents dealing with the same subtask typically learn similar policies and thus can share their training experiences to accelerate training. However, this method poses two new problems: (1) how to decompose multi-agent task into subtasks and (2) how to assign agents to subtasks. Most of the previous works [22 predefine the task decomposition by using a rich prior knowledge of application domains, which may not be available in practice. To the best of our knowledge, only one recent work, RODE [26). learns explicit task decomposition without prior domain knowledge. RODE defines the subtasks based on joint action space decomposition during pretraining. Specifically, RODE learns effect-based action representations to cluster actions and then treats each cluster of actions functionality of a subset of actions. But it may fail to solve subtasks when some basic actions are necessary for all subtasks, such the actions of moving in the StarCraft II micromanagement tasks (27). What's 's more, the effect of each action changes dynamically with the environment and thus may be difficult determine the actions' effect only through pretraining.
subtask, where each subtask only needs to fulfill the
In this work, we propose a novel framework to learn dynamic subtask assignment (LDSA) in cooper- ative MARL. Our method first proposes a subtask encoder that constructs vector representation for each subtask according to its identity. The action-observation history typically reflects the behavioral habits and potential abilities of each agent, which is an important clue for selecting subtasks. There- fore, we employ a trajectory encoding network to obtain the action-observation history of each agent. Then, for every timestep, each agent acquires categorical distribution of subtask selection based on the cosine similarity of its action-observation history and representations of all subtasks, and samples subtask using Gumbel-Softmax (28) for training, which reparameterization trick allowing backpropagate through samples. After that, we learn a separate policy for each subtask and enable agents dealing with the same subtask to share their learning. To associate the policy of each subtask with its representation, we introduce a subtask decoder to generate the policy parameters of each subtask based on its representation, which can also avoid similar policies between different subtasks. Furthermore, we introduce two regularizers to increase the representation difference between subtasks and avoid agents changing subtasks frequently to stabilize training, respectively.
We evaluate our method on the challenging StarCraft micromanagement tasks (SMAC) 27). The results show that our LDSA significantly improves the learning performance on the SMAC benchmark compared to the baselines, especially on Hard and Super Hard scenarios. Additional experiments on Google Research Football (GRF) [20] further demonstrate the effectiveness of LDSA on various multi-agent tasks. The ablation studies confirm the benefits of the two regularizers and the high efficiency of LDSA. Moreover, visualizations on SMAC reveal that LDSA could reasonably and effectively decomposes the task with dynamic subtask assignment for better collaboration.
2 Preliminaries
In this section, we introduce the necessary background knowledge to understand this paper. We first describe the problem formulation and the definition of subtasks for a fully cooperative multi-agent task. Then, we present the value function factorization methods with the centralized training with decentralized execution (CTDE) paradigm [2] 30), which will be adopted in this paper.

---

2.1 Problem formulation
This work is focused on a fully cooperative multi- task with only partial observation for each agent. This task is typically modeled a decentralized partially observable markov decision process (Dec-POMDP) 61 defined by a tuple = (A, U.P.P rr, Z, O,7), where A { 92. 9n. denotes the finite set of agents, 7 [0, 1) is the discount factor and 5 is the of global state of the environment, At each timestep. each agent Ja E A selects an action thg E U, forming a joint action transition
UH where according to the state transition function P(s'ls. Js, u) S x U x S + [0, 1] and all agents receive a S describes the global state. Due the partial
21, . is the agent identity. This leads
the environment
shared team reward r(s, u) S U + I. where observability, each agent Ja only obtains a partial observation Oa E Z according to the observation function ((s. 9a) S x - Z and holds an obstrvtibservttion history Ta E T (Z x U).. The joint action-observation history of all agents is denoted as T.
To achieve scalability, popular approach adopted by many recent MARL works 4 is that each agent Ia constructs a decentralized policy Ta (ua Ta; 0) with shared parameters 0. However, such simple parameter sharing may fail to deal with many complex multi-agent tasks that require diverse abilities among agents. To this end, we propose to decompose a fully cooperative multi-agent task into subtasks. We present the definition of subtasks in the following.
Definition 1 (Subtasks). For a fully cooperative multi-agent task = (A,, 2, we assume there exists a set of k subtasks, denoted as 4 = f$1, .2,-.-, ... and we consider
.b)), where k E N is unknown tunable kyperparameter. Each subtask 6i is defined by a tuple (E G Tod. where i E f1,2.... k is the identity of subtask, Ed.. E I is vector representation (or latent embeddings) of subtask Φi, Go. = (Aa., S, U, P. r.r, Z, 0,0,7) is the Dec-POMDP model of subtask ds.
and TØ; -T x U + 10 is the policy of subtask 6;. AØ; is the set of agents assigned to subtask os and each agent can only select one subtask to solve at each timestep. i.e., U As. and As. N = Ø i + j. Each agent 9a E Ag, shares the policy parameters of Té..
Our objective is to learn optimal set of subtasks 6= so as to maximize the expected global return 2 (T, u) E - Y r s, Ug = u, L). where r; the team reward at timestep t. Therefore, we are required to learn the vector representation Te.. the agent assignment As, and the policy Té. for each subtask d;, which will be introduced in detail in Sec.
2.2 Value function l'actorization with CTDE paradigm
To make policy learning stable and scalable in MARL, the paradigm of CTDE (2 30 has been proposed and gained substantial attention. In this paradigm, agents learn their policies together with access to global information during training in a centralized way and only rely on their local observations during execution in decentralized way. The CTDE paradigm has been exploited by among which value function factorization methods 4 challenging cooperative multi-agent tasks (27). These
many recent MARL algorithms 4
g have shown superior performance methods factorize the global Q-value function Qtat into individual Q-value functions Qa for each agent Ja. where Q. is only based on the local action-observation history тa for decentralized execution. To guarantee the consistency between greedy action selection in global and individual Q-values, such factorization has to satisfy andividual-Global -ooball-aaiiimmnn (IGM) principle (6] as follows:
arg max (T, arg max Q1 (7), uy)... (T), Mi
arg max Qu (Tny Un)) un
(1)
In this work, we consider representative value function factorization method, QMIX 5 which estimates global Q-value as non-linear combination of individual Q-values as follows:
QToe(T, u) = fw (Qi(T, u1), ... Qn.nn: us))s),
(2)
where fw. is monotonic function that conditions the global state s, i.e., aQ_
8f > 0, Va
(1, 2...... m.]. Although Eq. is is only sufficient and unnecessary IGM, QMIX a lightweight and efficient method showing offte-of-the-art performance SMAC [27].
3

---

Ozehot
Qtor
Subtask Encoder
Gumbel Softmax Sampling
x
Qa
Mixing Network
x
Q-n
Axa
X FC
Subtank Os policy
Subtask Φ policy
Subtask policy
Trajectory Encader
Qastr
FC
GRU
ha
hg
GRU
h3
Subtask Decoder
RC
Oa
0g
(a)
(b)
(c)
Figure I: Overview of LDSA framework. (a) The subtask representation learning structure (shown in blue). (b) Architecture of subtask selection for each agent (shown in green). (c) The policy learning for each subtask (shown in purple).
Method
In this section, present the LDSA learning framework shown in Fig. 1 We first introduce how to construct set of distinct subtasks for decomposing a multi-agent task. Next, we discuss how each agent selects a subtask according its abilities. After grouping agents by subtasks, we present the policy of each subtask that is representation-deeendee. Finally, we show the overall training objective and the inference strategy.
3.1 Distinct subtask representation
Many complex multi-agent tasks involve set of subtasks that have different responsibilities. For example, running company requires multiple departments work together. Previous work typically leverages the prior domain knowledge to decompose a complex task, which not practical for many uncertain environments. To remove the dependence prior knowledge and apply it to broader multi-agent tasks, we propose to learn a vector representation Es. for each subtask Φ; according to its identity i.
Specifically, we use two-layer fully-connected (FC) network with a tanh activation function in Fig. aa to learn subtask encoder fe(:;0a) TR - Rm parameterized by 0g. The subtask encoder maps the one-hot identity d, E R²* of subtask Φ; to an m-dimensional representation space. The tanh activation function constraint the value range of the representation space. Then, we employ the subtask encoder to construct vector representation ZΦ. R. for each subtask Φ;, i.e.
Is. = feld;; (1,2,-...
If subtasks are similar, the task decomposition makes little sense. Therefore, keep differences between subtasks, we propose regularizer that maximizes the L2 distance between representations of different subtasks as follows:
Co(0s) EED
Σ
10 Ie, 112]
where D is the replay buffer. The subtask representation learning continues throughout the training process, which can automatically adapt to the dynamic changes in the environment. The overall optimization objective of the subtask encoder will be introduced in Sec. 3.4
3.2 Ability-based subtask selection
With the representation of subtasks, we need to design subtask selection strategy for each agent based on its ability. In real life, can easily infer a person's 's role from the trajectory of his or her

---

behavior. Similarly, the action-observen observation history of an agent can reflect its behavioral habits and potential abilities. Therefore, we utilize a shared trajectory encoder (shown in Fig. (Tb) consisting of GRU [32 and two fully-connected networks to obtain the action-observation history of each agent. The trajectory encoder (:; 0n). parameterized by 0,, encodes the action-observation history of each agent Do into a vector r. E TRm which has the same length with the subtask representation. Then, agent Oa treats Ir, as its ability representation and selects a subtask to solve based on Ir.-
In our implementation, for each agent Ja. we first calculate the cosine similarity of its action- observation history representation Zr, and representations of all subtasks Z4 := [ i.e. simdlaiily(arr, ) = (xc), xt.)) [[[= [I]|ara, I]). Since representations of all subtasks have the same value range through the tanh activation function, we use xt_ Jo, to approximate the cosine similarity for simplicity. Then, we employ softmax function on the cosine similarity and obtain a categorical distribution of subtask selection p[0|xT=> ;T,, xTp) == [p(0, "4 p( Igp) is the probability that agent In selects subtask 6i. given by:
exp(x Fet))
p(0:|TTT TO)
Vi E (1,2,...,
(5)
Σラ= exp((x? Tbz)
where exp(.)) is the exponential function. Sampling subtask directly from the categorical distribution p( (4.)) is not differentiable. To make the subtask selection process trainable, we use the Ttrought-Through Gumbel- Estimator [28] to sample subtask Φ, that will be discretized as a k-dimensional one-hot vector, i.e., the one-hot subtask identity dz.
For every timestep, each agent will select subtask to dynamically cooperate with each other. This dynamic subtask assignment scheme may make training unstable when an agent frequently changes the subtask in an episode. To smooth the agents' subtask selection and stabilize training, we introduce a second regularizer to minimize the KL divergence between the subtask selection distributions for any two adjacent timesteps as follows:
(n(00,0)) EE Σ DKL (p(1 ar,, I (lll اbم)
(6)
where p'("'l<rr,, s(bb) is the subtask selection distribution at the next timestep, (-I(-1l)) is the KL. divergence operator and the sum is performed across all agents.
3.3 Representation-deeendde subtask policy
After grouping agents to different subtasks based on their abilities, we learn the policy for each subtask which is illustrated in Fig. [c). Generally, agents dealing with the same subtask share the policy parameters and different subtasks have distinct policy parameters. To this end, we first use new shared trajectory encoder f=(100,) with parameters 0, to generate the action-observation history of each agent 9a. denoted as hy., The new trajectory encoder f- (0,) only consists of a fully-connected network and a GRU, The policy of each subtask o; is fully-comnected network for (:: with parameters 0s.. Then, for each agent 9a E As, that solves the subtask Φ., we feed its action-observation history hr into the subtask policy fe, (=:0g, and generate the individual Q-value Qu- To associate the policy of each subtask with its representation, we utilize a subtask decoder fa(-10a) conditioning on the subtask representation to generate the policy parameters of each subtask. The subtask decoder fa(- (2) with parameters 6g is just a single-layer fully-connected network, which maps Táz to Os, for each subtask di By virtue of the differences between subtask representations, such subtask decoder can further increase the diversity of subtask policies.
In our implementation, for each agent 9a. we feed hr, into all subtask policies to get the individual -aluuu Qa.e, for every subtask Φ, The individual Q-value Qa is the sum of @2 weighted by the one-hot identity of the selected subtask. If agent Ja selects subtask dj. Qa is actually equal to Qa.s, Hence, each agent only trains the policy parameters of its selected subtask. In this way. agents with similar abilities tend to select the same subtask and thus can share their experiences to accelerate training and improve performance. Moreover, on the tasks that require to assign different subtasks to agents with similar trajectories, since each agent's observation input contains its one-hot identity, LDSA could learn to put more weights on the agent's identity input, and make the agent's identity
5

---

input become the main factor of the output subtask selection distribution. Then two agents can have distinct subtask selection distributions even if they have similar observation from the environment.
3.4 Overall training and inference
Similar value function factorization methods 8 ow to map all agents' individual Q-values (Qa. Q - into the global Q-value Qiot. where Qa the individual Q-value of agent Ia and Q-a are the individual Q-values of other agents. In this work, we adopt the mixing network introduced by QMI [5] thanks to its simple structure and superior performance. can be easily extended other more complex mixing networks 77 Then, all the parameters = (fe, On, 0r, 0d, (w) our framework can be optimized by minimizing the TD loss of Qtot as follows:
mixing network with parameters
(O) ED (1+ mmx Qror ,u;00) Qtor (T, u; m))))
where 0- are the parameters of the target network that periodically copied from 0. Considering the two regularizers in Eq. 4 and Eq. [6] the overall optimization objective of LDSA is:
(()) = (i) + (te) + (fe.
8
where Ag and X are positive coefficients of the two regularizers, respectively. During the inference phase (i.e., test the decentralized policy), each agent Ia selects the subtask with the maximum probability on the subtask selection distribution, i.e., arg max;, p zg) ) and then chooses a greedy action according the individual Q-value Qa for decentralized execution.
Experiments
In this section, conduct several experiments to investigate the following questions: (1) Can LDSA improve the performance compared to the baselines? (Sec. (42) (2) How do the two proposed regularizers influence the performance? (Sec. (.3) (3) Whether the superior performance of our method comes from the increase in the number of parameters? If not, which component contributes the most to method? (Sec. (4.3) (4) Can LDSA learn dynamic subtask assignment and group agents reasonably? If so, agents with similar abilities solve the same subtask and each subtask has specific responsibility. (Sec. 4.4)
4.1 Experimental setup
Environment We evaluate LDSA on the SMAC benchmark (27). challenging benchmark for cooperative MARI. There are two armies of units in the SMAC environment, Each ally unit controlled by decentralized agent that can only act based on its local observation and the enemy units are controlled by built-in handcrafted heuristic rules. The goal of the MARI. algorithm is to maximize the test win rate for each battle scenario. In this paper, settings for SMAC. The version used in this work is SC2.4.10. 10. We consider all 14 scenarios on the SMAC benchmark that can be classified into three different levels of difficulties: Easy, Hard and Super Hard scenarios.
adopt the default environment
Baselines We select QMIX ROMA [33 and RODE 26 as our baselines. All of the baselines and LDSA belong to value function factorization methods and use the QMIX-style mixing network [5) for fair comparison. Therefore, LDSA differs from the baselines only by the individual Q-value networks. QMIX is natural baseline in which all agents share the individual Q-value network. ROMA and RODE are two recent works that learn implicit and explicit task decomposition in MARL, respectively. We implement all the baselines using their open-source codes based on PyMARL [27]. Note that, our experiments are conducted under the assumption that there no prior knowledge about the environment. As mentioned in Sec. [ RODE may not work when some basic actions are necessary for subtasks. the SMAC environment, there are four actions representing the agents' movement in four cardinal directions, which are important for all subtasks. To be effective,

---

corrdor
3652 V9_
6t_
100
LDSA
Median leet Vint Rete(%)
Median Vinn Rater%)
ROMA
60
60
15
40
Modinn
20
20
0
0.0
1.0
2.0
3.0
4,0
5.0
0.0
1.0
2.0
3.0
4.0
5.0
0.0
1,0
2.0
3.0
4.0
5.0
(imestepe(k))
Timesteps(M)
Timesteps(M)
Smn. ve. Bm
2c, ve. 642g
3e. ve. .2.
Medan Rater())
Median Weit Rater%)
Median Test Winn Raterk)
80
80
80
60
20
20
20
0.0
0.5
1.0 Timesteps(M)
1.5
2.0
0.5
1.0
1.5
2.0
0.0
1,0
1.5
20
TimestepsMM)
Timesteps(M)
Figure 2: Comparison of our method against baselines three Super Hard SMAC scenarios: corridor, 3s5z_ve vs vs_3s6z, vh_vs_8z and three Hard SMAC scemarios: 5m_vs, soo_v_6ee 6m, _o__a_6442g 3s_vs_52 5z. The solid line shows the median test win rate across 5 seeds and the shaded areas correspond to the 25-75% percentiles.
RODE several nules based prior knowledge of the environment in its implementation, which manually enforce the four actions of moving be available to some subtasks forever under certain conditions. Therefore, not involve prior knowledge, we remove these rules when we implement RODE. We use "RODE_n nr" to represent RODE without manual rules in our experiments.
Hyperparameters For all experiments, the number of subtasks is set to 4 and the length of subtask representations set to 64, ie., k 4, m 64. We carry out grid search for regularizers' coefficients A and A on the SMAC scenario corridor and then set them to 10-3 and 10-3, respectively, for all scenarios. All the common hyperparameters our method and baselines are set to be the same as that in the default implementation of PyMARL. We use lightweight network structures for the subtask encoder, subtask decoder, trajectory encoder and subtask policy. The detailed hyperparameters and network structures will be provided in Appendix A.
4.2 Performance on SMAC
We compare LDSA with the baselines across all 14 scenarios on the SMAC benchmark (27). For every scenario, we carry out 5 independent runs with 5 different random seeds for all methods and show the median performance and 25-75% percentiles. Fig. 3 presents the averaged median test win rate across all 14 scenarios. We can observe that LDSA significantly improves the learning performance and surpasses about 7% median test win rate averaged across all 14 scenarios.
Fig. 2 shows the learning curves LDSA and the base- lines six Hard or Super Hard scenarios. The results on the other eight scenarios are shown in Appendix B. LDSA outperforms all the baselines all Super Hard
Mosmggan Madaan root-Wn Rato(%)
75
LDSA
60
QMIX
ROMA
RODE
and Hard scenarios with faster convergence. Specifically, on the scenarios that require diverse micromanagement techniques: corridor, 3s5z_vs vs_3 and 6h_vs_B vs_8z, the test win rate of LDSA exceeds that of the baselines at least 15%. On the Easy scenarios, LDSA shows similar performance as QMIX, which indicates that our method may not improve the learning performance on simple tasks not requiring task decomposition. Moreover, ROMA that learns implicit task decomposition performs even worse than QMIX on most the scenarios. Without using rules based prior knowledge, RODE fails to learn efficient
0.0
0.5
1.0
1.5
2.0
Timesteps(M)
Figure 3: The averaged median test win rate across all 14 scenarios on the SMAC benchmark.

---

CO/roor
5mn vo_ 8m
20_ vo. 642g
100
100
100
LDSA
Median leest Vilt Rete(%)
Median Tiest Velt Rate(%)
Median Vein
60
LDSA NR
NN
60
60
40
20
20
0
0.0
1.0
2,0
3.0
4.0
5.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0 TimestepsMM)
1.5
2.0
Timesteps(M)
Timesteps(M)
Figure 4: Ablation studies regarding the two proposed regularizers. "NP" and "NR" mean to remove La and Lg from the overall optimization objective of LDSA, respectively. The best performance of QMIX is shown as a dashed line.
conrdor
5m,, vs. 8m
2t, vs. 642g
Medan VeotWin Rater()
Modian Thss Tist Rate(S)
Median Test Win Rater%)
80
LDSA N.Desakks
80
80
RasSale
60
UMIX Large
Win
20
20
20
0
0.0
1.0
2.0
3.0
4.0
5.0
0.5
1.0
1.5
2.0
0.0
1.5
20
TimestepsjM)
Timesteps((1)
Timesteps/M)
Figure 5: Ablation studies regarding components of LDSA. "LDSA_ NoDecoder" represents LDSA without the subtask decoder. "LDSA RanSele" indicates that LDSA randomly selects a subtask for each agent at each timestep. "QMIX Large" means to increase the number of parameters in QMIX to be similar as that in LDSA. The best performance of QMIX is shown as a dashed line.
"QMIX_Large"
policies for subtasks, which demonstrates that discovering subtasks based on joint action space decomposition and restricting the action space of subtasks may severely limit the learning of subtask policies. These results further confirm that LDSA can learn effective task decomposition with dy- namic subtask assignment to solve complex tasks. In Appendíx D, compare LDSA with another baseline CDS [16] that celebrates diversity among agents, and show the benefits of LDSA to balance the training complexity and the diversity of agent behavior. We also evaluate LDSA on Google Research Football (GRF) 20 to demonstrate the effectiveness of LDSA on various multi-agent tasks in Appendix E.
4.3 Ablation studies
In this subsection, we conduct ablation studies on three scenarios: corridor, 5m_vs_ vs_6m and 2c_vs_ cc_vs_64zg, to show the effect of the two regularizers and test which component contributes most to LDSA. Fig. reports the performance of LDSA when we remove each or both of the two regularizers Ln and L.- can be observed that the performance of LDSA without both of the two regularizers is even worse than that of QMIX. Either of the two regularizers can improve performance and L. improves more than La which indicates that keeping differences between subtasks is more important while LA. can stabilize training.
To figure out why LDSA performs better than QMIX, investigate the contributions of three components of LDSA; the increase in the number of parameters, the ability-based subtask selection and the subtask decoder. As shown in Fig. 5 QMIX with similar parameters LDSA can't t improve performance and even may make training slower and degrade performance, which proves that the superior performance of LDSA is not due to the increase in the number of parameters. The performance of LDSA without the subtask decoder is lower than that of LDSA, which demonstrates that associating subtask policy with subtask representation benefits the performance. What's more, the large margin between the performance of LDSA with random subtask selection and that of LDSA reveals that the ability- subtask selection contributes the most to our method. In summary, the superior performance of LDSA is largely due to the efficient subtask assignment. We also study the effect of the number subtasks in Appendix C and conduct two more ablations for LDSA in Appendix F.
8

---

subteek A
subteek 42
subteekc d
subtask (4,
63
64 114
Figure 6: Visualizations of dynamic subtask assignment in one episode (114 timesteps) on corridor Different colors indicate different assigned subtasks. (a) The subtask assignment for six ally agents along the whole episode. (b) and (c) show the game screenshots t = and = 64, respectively, where each ally agent is marked by colored number. The number represents the agent identity and the color indicates the assigned subtask.
4.4 Visualization of dynamic subtask assignment
In this subsection, we visualize the dynamic subtask assignment in one episode on the SMAC scenario corridor as shown Fig. 6 The scenario corridor consists of 6 ally Zealots and 24 enemy Zerglings. Due to the huge disparity in the number of allies and enemies, it's hard to win if all ally Zealots rush to attack the enemy. Therefore, at the beginning (f = 3), our method learns to sacrifice two allies (agents 91 and 94) to attract most enemies and then the other 4 allies (agents 92, I3. 9r and 96) can focus fire to kill a small part of enemies. After that, the remaining enemy Zerglings controlled by heuristic rules will go through narrow corridor, and stay the bottom left corner of the scenario there are no Zealots within the sight range. In the middle the episode (t = 64), the remaining enemy forces still strong, number of enemies and kill them with the other allies (agents 93. 95 and go). This process will repeat until all enemies and 94) to a fixed subtask (subtask dg),, which can prevent them from interfering with the learning of other subtasks. Besides, is worth noting that subtask Φ1 and subtask Φ4 have similar responsibilities that agents focus fire to kill enemies, but under different situations related to the agent's health point. When an agent needs focus fire kill enemies, tends to select subtask Φi if has a good health point otherwise select subtask Φ4. These visualizations demonstrate the effectiveness and rationality of the dynamic subtask assignment our method.
method assigns the healthiest ally (agent g2) to draw out small
killed. Moreover,
find that method will assign the dead allies (agents gi
Related work
Value-based MARL Value-based MARL algorithms have achieved great progress in recent years. IQL [34] simply trains an independent Q-value network for each agent, which treats the other agents as part of the environment. This method may not converge due to the non-stationarity of the environment caused by the changing policies of other agents. The other extreme method is to treat all agents as single agent and learn a global Q-value based the joint action-observation space, which alleviates the non-stationarity but suffers from the scalability challenge, as the joint action- observation space grows exponentially with the number of agents. To trade off these methods, most of the value-based MARL algorithms factorize the global Q-value into individual Q-values for centralized training and decentralized execution. VDN (4) and QMIX (5) factorize the global Q-value by additivity and monotonicity, respectively. QTRAN 6 transforms the original global Q-value function into an easily factorizable one expand the representation capacity. QPLEX [8] proposes duplex dueling network architecture to implement the complete IGM [6] function class. These methods focus on designing the mixing network. There are other value-based works studying
MARL from the perspective of communication [35 36), exploration [7 38 and robustness (39 40
Parameter sharing in MARL To learn efficiency and scalability, most of the MARL works employ the technique of parameter sharing among agents [4[5]. Terry et al. [17] demonstrate that parameter sharing can effectively alleviate the non-stationarity problem MARL. Despite its efficiency, fully-shared parameters among agents may destroy the diversity of agent behavior in many complex multi-agent tasks 20 27.. To trade off experience sharing and behavioral diversity among agents, SePS [21 groups agents based their identities during pretraining and each group

---

shares one policy for training. CDS [16] of a shared part and non-shared part. In this work, we don't fix the group each agent can share with. Each agent in our method can adaptively choose group to share based on its trajectory during training, which enables dynamie parameter sharing.
proposes to decompose the policy of each agent as the sum
Task decomposition in MARL Task decomposition plays an important role in many complex real-world multi-agent systems, such as software engineering [41], healthcare [42] and traffic man- Once the multi-agent task decomposed, agents can be assigned to the restricted subtasks that are easier to solve, and thus the learning complexity is greatly reduced, However, it is challenging to come up with set of subtasks that can effectively decompose the whole multi-agent task. The most straightforward way is to predefine the subtasks by leveraging the prior domain knowledge (22-25). But the prior knowledge may not be available in many uncertain environments. To solve this problem, ROMA [33 introduces the concept of roles for each agent based on its local observation and conditions agents' policies on their roles. Nguyen et al. 44 redesign the first layer of the mixing network in QMIX [5] and view the output nodes of the first layer as different roles. These two works focus on implicit task decomposition. RODE (26) explicitly defines the roles/subtasks based on joint action space decomposition during pretraining, where each subtask corresponds to subset of actions. In this work, we learn vector representations for subtasks that can automatically adapt the environment during training.
agement 4
6 Conclusion
Task decomposition is an important approach to simplify complex multi-agent tasks and has not been well solved without using prior knowledge. To this end, we propose decompose the task into several subtasks represented by latent embeddings. Agents select subtasks according to their abilities, which are indicated by their ration-observatton histories. In this way, agents dealing with the same subtask can share their learning to solve the subtask, which can learn the specific abilities required by all subtasks under tractable training complexity. Although the embedding representation for each subtask may be abstract, it essentially clusters agents with similar abilities into the same group and thus agents can dynamically share their experiences to accelerate training and improve performance. After training, each agent is aware of all subtask policies and can adaptively choose the most appropriate subtask policy to execute based on its ability. The empirical results further demonstrate that LDSA learns effective task decomposition with dynamic subtask assignment and significantly improves the learning performance compared to the baselines. We hope our method could provide a new perspective on task decomposition and subtask learning in MARL.
Acknowledgments
This work was supported by the National Natural Science Foundation of China under Contract 61836011 and 62021001, and by the Huawei Cloud project "Multi-agent Competitive Decision Scenario Algorithm and Technology Research". was also supported by GPU cluster built by MCC Lab of Information Science and Technology Institution, USTC.
References
[1] T. Wu, P. Zhou, K. Liu, Y. Yuan, X. Wang, H. Huang, and D. O. Wu, "Multi-agent deep reinforcement learning for urban traffic light control in vehicular networks," IEEE Transactions on Vehicular Technology. vol. 69, no. 8, pp. 8243-8256, 2020.
[2] Y. Cao, W. Yu, W. Ren, and G. Chen, "An overview of recent progress in the study of distributed multi-agent coordination," IEEE Transactions on Industrial informatics, vol. 9, no. pp. 427-444444 438, 2012.
[3] M. Hüttenrauch, A. Šošić, and G. Neumann, "Guided deep reinforcement learning for swarm systems," arXiv preprint
2017.
[4] P. Sunchag, G. Lever, A. Gruslys, W. M. Czamnekki, V. Zambaldi, M. Jaderberg, M. Lanctot, N. Sonnerat, J. Z. Leibo, K. Tuyls et al., saaue-ddccompsittion networks for cooperative multi- agent learning." arXiv preprint arXiv: 05296, 2017.
10

---

[5] T. Rashid, M. Samvelyan. C. Schroeder, G. Farquhar. J. Foerster. and S. Whiteson, "Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning." in Proceedings of the International Conference on Machine Learning. PMLR, 2018, pp. 4295-4304.
[6] K. Son, D. Kim, W. J. Kang, D. E. Hostallero, and Y. Yi. "Qtran: Learning factorize with transformation cooperative multi-agent reinforcement learning" Proceedings of the International Conference on Machine Learning. PMLR, 2019, pp. 5887-5896.
[7] Y. Yang. J. Hao, B. Liao, Shao, G. Chen, W. Liu, and Tang, "Qatten: general framework for cooperative multiagent reinforcement learning," arXiv preprint arXiv:2002.088 030399, 2020.
[8] J. Wang, Z. Ren, T. Liu, Yu, and C. Zhang, "Qplex: Duplex ducling multi-agent q-learning," " in Proceedings of the International Conference on Learning Representations, 2021.
[9] S. lqbal, A. S. De Witt, Peng, W. Böhmer, S. Whiteson, and F. Sha, "Randomized entity- wise factorization for multi-agent reinforrmeen learning"" " in Proceedings of the International Conference on Machine Learning. PMLR, 2021, pp. 4596-4606.
[10] R. Lowe, I. Wu, A. Tamar, J. Harb, O. Pieter Abbeel, and I. Mordatch, "Multi-agent actor-critic for mixed cooperativ-competiive environments," in Proceedings of the Neural Information Processing Systems, vol. 30, 2017.
[11] J. Foerster, G. Farquhar, T. Afouras, N. Nardelli, and S. Whiteson, "Counterfactual multi-agent policy gradients." in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32, no. 1, 2018.
[12] M. Zhou, Z. Liu, P. Sui, Li, and Y. Y. Chung, "Learning implicit credit assignment for cooperative multi-agent agent reinforcement learning." in Proceedings of the Neural Information Processing Systems, vol. 33, 2020, pp. 11 853-11 864.
[13] Y. Wang. B. Han, T. Wang, H. Dong, and C. Zhang, "Dop: Off-policy multi-agent decomposed policy gradients," in Proceedings of the International Conference Learning Representations, 2020.
[14] J. Su. S. Adams, and P. A. Beling, "Valuedeeoomposin multi-agent actor-critics," in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, no. 13, 2021, pp. 352-11 360.
[15] T. Zhang. Y. Li, C. Wang, G. Xie, and Z. Lu, "Fop: Factorizing optimal joint policy of maximum- eeximummentropy multi-agent reinforcement learning"" in Proceedings of the International Conference on Machine Learning. PMI.R, 2021, pp. 12 491-12.0000.
[16] L. Chenghao, T. Wang, C. Wu, Q. Zhao, J. Yang, and C. Zhang, "Celebrating diversity in shared mulli--geet reinforcement learning." in Proceedings of the Neural Information Processing Systems, vol. 34, 2021.
[17] J. K. Terry, N. Grammel, A. Hari, and L. Santos, "Parameter sharing is surprisingly useful for multi-agent deep reinforcement lcarning," arXiv preprint arXiv:2005. 13625, 2020.
[18] H. Tiantield, J. Tian, and X. Yao, "On the architectures of complex multi agent systems," " in Proceedings of the Workshop on Knowledge Grid and Grid Intelligence. Citeseer, 2003, pp. 195-206.
[19] C. Witteveen and M. De Weerdt, AMulti-agemi planning for non-cooperative agents." in Proceedings of the AAAI Spring Symposium: Distributed Plan and Schedule Management, 2006, p. 169.
[20] K. Kurach, A. Raichuk, P. Stanczyk, M. 7ajac, O. Bachem, L. Espeholt, C. Riquelme, D. Vincent, M. Michalski, O. Bousquet al., "Google research football: novel reinforcement learning environment," in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, no. 04, 2020, pp. 4501-4510
[21] F. Christianos, G. Papoudakis, M. A. Rahman, and S. V. Albrecht, "Scaling multi-agent reinforcement learning with selective parameter sharing," in Proceedings of the International Conference on Machine Learning. PMLR, 2021, pp. 1989-1998.
[22] Pavón and J. Gómez-Sanz. "Agent oriented software engineering with ingenias," in Proceedings of the International Central and Eastern European Conference on Multi-Agent Systems. Springer, 2003, pp. 394 403.
[23] M. Cossentino, S. Gaglio, L. Sabatucci, and V. Seidita, "The passi and agile passi mas meta-models compared with a unifying proposal," in Proceedings of the International Central and Eastern European Conference on Multi-Agent Systems. Springer, 2005, pp. 183-192.
[24] N. Spanoudakis and P. Moraitis, "Using aseme methodology for model-driven agent systems development." in International Workshop Agent- Oriented Software Engineering. Springer, 2010, pp. 106-127.
11

---

[25] N. Bonjean, W. Mefteh, M.-P. Gleizes, C. Maurel, and F. Migeon, "Adelfe 2.0, handbook on agent-oriented design processes," 2014.
[26] T. Wang, T. Gupta, A. Mahajan, B. Peng, S. Whiteson, and C. Zhang, "Rode: Learning roles to decompose mulli-agent tasks," Proceedings of the International Conference on Learning Representations. 2021.
[27] Samvelyan. T. Rashid, C. S. De Witt, G. Farquhar. Nardelli, T. G. Rudner, C.-M. Hung, P. H. Torr. J. Foerster, and S. Whiteson, "The starcraft multi-agent challenge." " arXiv preprint urXiv:1902.04044 2019.
[28] E. Jang, S. Gu, and B. Poole, "Categorical reparameterization with 1rXiv:6611.11444 2016.
arXiv preprint
[29] F. A. Oliehoek, M. T. Spaan, and N. Vlassis, "Optimal and approximate q-value functions for decentralized pomdps," Journal of Artificial Intelligence Research, vol. 32, pp. 289-353, 2008
[30] L. Kraemer and B. Banerjee, "Multi-agen agent reinforcement learning as rehearsal for decentralized planning." Neurocomputing, vol. 190. pp. 82 94, 2016.
[31] F. A. Olichoek and C. Amato, A concise introduction to decentralized POMDPs. Springer. 2016.
[32] K. Cho, B. Van Merriebbeer C. Gulcchre, Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio, "Learning phrase representations using mn encoder-decoder for statistical machine translation," arXiv preprint 1rXiv:1406. 1078, 2014.
[33] T. Wang, H. Dong, V. Lesser, and C. Zhang, "ROMA: Multi-agent reinforcement learning with emergent roles," in Proceedings of the International Conference Machine Learning. PMLR, 2020, pp. 9876- 9886.
[34] A. Tampuu, Matiise,, D. Kodelja, I. Kuzovkin, K. Korjus, J. Aru, J. Aru, and R. Vicente, "Multiagent cooperation and competition with deep reinforcement learning," " PloS one, vol. 12, no. 4, p. e0172395, 2017.
[35] J. Foerster, I. A. Assael, N. De Freitas, and S. Whiteson, "Learning to communicate with deep multi-agent reinforcement learning." in Pruceedings of the Neural Information Processing Systems, vol. 29, 2016.
[36] T. Wang, J. Wang, C. Zheng, and C. Zhang, "Learning nearly decomposable value functions via com- munication minimization,"" in Proceedings of the International Conference on Learning Representations, 2020.
[37] A. Mahajan, T. Rashid, M. Samvelyan, and Whiteson, "Maven: Multi-agent variational exploration," in Proceedings of the Neural Information Processing Systems, vol. 32, 2019.
[38] T. Gupta, A. Mahajan, B. Peng. W. Böhmer, and Whiteson, "Uneven: Universal value exploration for multi-agent reinforeement learning," in Proceedings of the International Conference on Machine Learning. PMLR, 2021, pp. 3930-390 3941.
[39] K. Zhang, T. Sun. Y. Tao, S. Gene, S. Mallya, and T. Basar, "Robust multi-agent reinforcement learning with model uncertainty," in Proceedings of the Neural Information Processing Systems, vol. 33, 2020, pp. 105 583.
[40] J. Zhao, Y. Zhao, W. Wang, M. Yang, X. Hu, W. Zhou, J. Hao, and H. Li, "Coach multi multi-agent reinforcement learning framework for unexpected crashed agents," arXiv preprint arXiv:203..08454 2022.
[41] D. Smolko, "Design and evaluation of the mobile agent architecture for distributed consistency man- agement," in Proceedings of the International Conference on Software Engineering. IEEE, 2001, pp. 799-800.
[42] T. Alsinet, R. Béjar, C. Fernanadez, and F. Manyà, "A multi-agent system architecture for monitoring medical protocols," in Proceedings of the International Conference on Autonomous Agents, 2000, pp. 499-505.
[43] F. Logi and S. G. Ritchie, "A multi-agent architecture for cooperative inter-jurisdictional traffic congestion management," Transportation Research Part C: Emerging Technologies, vol. 10, no. 5-6, pp. 507-527, 2002.
[44] D. Nguyen, P. Nguyen. S. Venkatesh, and T. Tran, "Learning to transfer role assignment across team sizes." arXiv preprint MxXiv2204.112377 2022.
12

---

Checklist
assets... 1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you descrihe the limitations of your work? [Yes] In Sec. 4.2 we mention that our method may not improve the learning performance on simple tasks not requiring task decomposition.
(c) Did you discuss any potential negative societal impacts of your work? [N/A] Although we haven't it evaluated our method on real applications, we believe our work doesn't have potential negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] We include the code, data, and instructions needed to reproduce the main experimental results in the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Sec. 4.1 and Appendix
(c) Did you report error bars (c.g., with respect to the random seed after running experi- ments multiple times)? [Yes] We report the median performance well as the 25-75% percentiles across random seeds.
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, cloud provider)? [Yes] See Appendix A.
4. If you using existing assets (e.g., code, data, models) curating/releusing
(a) If your work uses existing assets, did you cite the creators? [Yes] We implement our method based on the open- sourced PyMARL framework, which has been cited in Sec. 1.11
(b) Did you mention the license of the assets? [Yes] PyMARL is licensed under the Apache License 2.0.
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "nukMwMhOPMl", "reviewer": "Reviewer_z7kN", "summary": "This work addresses skill/ability specialisation for agents sharing a controller in cooperative MARL. LDSA, the proposed approach in this work, assumes agents' skill specialisation is required to solve subtasks constituting a particular task. LDSA subsequently learns to assign agents to subtasks by:\n1) Creating identically sized representations of subtasks and agents' abilities.\n2) Match agents and subtasks according to the similarity of their representations.\n3) Learning a policy that agents need to follow for each subtask.\n\nThe process of learning representations of agents & subtasks, alongside the optimal policy for each subtask, is done end-to-end in combination with QMIX, which is a value-based value factorisation MARL method. Aside from the temporal difference loss functions associated with QMIX, LDSA's optimised function also includes regularisation terms that (a) force subtasks to have different representations from each other and (ii) agents to have similar subtask assignment distributions between consecutive timesteps.\n\nThis work subsequently evaluates LDSA in various scenarios in SMAC. LDSA's performance is compared against QMIX and previous MARL methods for role discovery, such as ROMA and RODE. This work also conducted an ablation study to find the importance of the QMIX-based objective function alongside other regularisation terms for the trained agents' returns. These experiments' results empirically demonstrate LDSA's significantly improved performance compared to previous role assignment methods. Further analysis of the resulting agents' behaviour in the SMAC corridor scenario indicates agents' ability to solve diverse tasks to perform well in the environment.", "questions": "**Questions**\n\n1. How does LDSA deal with the need for assigning different roles to agents with similar observed trajectories?\n2. How will LDSA perform compared to a method that directly estimates the mixture weights of each subtask policy as opposed to computing it based on agent ability-subtask representational similarity.\n3. What happens if LDSA uses a mixture of subtask policies (that are weighted according to inferred component weights) as opposed to sampling one subtask policy at any timestep?\n4. In Figure 6 on Section 4.4, what is the difference between the subtask denoted by the red and blue lines?\n5. In the problem formulation, why does subtasks have separate transition function compared to the solved task? Isn't it the case that tasks and subtasks are situated in the same environment (and hence sharing the same transition function)?\n\n**Minor suggestions for writing**\n\n1. Consider using different names for the two trajectory encoders in Lines 160 and 185.\n2. For each environment used in evaluation, consider mentioning the different roles that are required for solving each environment. ", "limitations": "As indicated by the feedback provided in the previous sections, the authors have not highlighted the potential limitations of their work. Furthermore, I believe no statement regarding potential negative social impact of the work is required since this work is currently a generic MARL method that has yet been applied to real-world problems.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Originality**\n\n**(Minor Weakness) LDSA is an incremental idea relative to the most recent works in the area.**\n\nThe idea of assigning agents to roles based on their observed trajectories has been explored before in the ROMA approach (Wang et al., 2020). While ROMA proposes an implicit role assignment method that conditions agents' policies on task representations, explicit approaches like LDSA that learn K separate policies for different subtasks and the optimal mixture of these policies has recently been explored by Nguyen et al. (2022). Compared to the idea from Nguyen et al. (2022), LDSA merely offers a different way of computing the weights and utilisation of policies of each subtask.\n\n**Quality**\n\n**(Major Strength) Comparison against several prior methods in role assignments show LDSA's significantly higher returns.**\n\nBy outperforming ROMA and RODE, this work demonstrates that LDSA has the potential of improving several existing approaches for role assignment. \n\n**(Major Strength) Empirical evaluation of learned subtasks.**\n\nEmpirical evaluation of learned subtasks in SMAC's corridor setup indicates that LDSA manages to learn meaningful subtask assignment in some environments. This experiment further illustrates the significance of this work's results and should ideally be done for all evaluated environments.\n\n**(Major Weakness) Missing baselines on skill/ability specialisation for parameter sharing in MARL.**\n\nSince the abstract and introduction section frames LDSA as a solution for skill/ability specialisation under parameter sharing in MARL, a comparison against CDS (Li et al., 2021) is necessary since it also attempts to achieve the same goal as LDSA. \n\nBeyond the similarity in their underlying goals, comparing LDSA against CDS may also help investigate the effects of one of LDSA's potential weaknesses. In particular, LDSA assigns a role to agents based on their observed trajectories. This can be a drawback when agents exhibiting similar trajectories must be assigned different subtasks to improve overall performance. For instance, consider a kitchen setup where two agents must clean dirty plates, chop different types of ingredients, and serve the chopped ingredients on the cleaned plates. Under this setup, the optimal policy is for both agents to jointly wash the dirty dishes, divide the chopping of different ingredients among them, and subsequently serve the ingredient they have chopped on a plate. Note that before ingredients are chopped, the trajectory of both agents is similar since they are solving the same task up to that point. This lack of a symmetry-breaking mechanism to assign agents with similar trajectories to different subtasks is not present in CDS since each agent in CDS has individual components not shared with others. \n\n**(Major Weakness) Missing baseline to highlight the importance of the proposed role assignment method.**\n\nSince LDSA main novelty is its proposed role assignment method, the baselines can be useful to highlight the importance of LDSA's role/subtask assignment components:\n\n1. A baseline where task assignment probability is directly estimated from agents' observed trajectories. Comparing LDSA's performance against this baseline helps us understand the importance of mapping subtasks & agent trajectories to fixed-length representations and computing their similarities to compute task assignment probability. \n2. A baseline where an agent employs a mixture of the individual subtasks' policies rather than using a single sampled policy.\n\nNote that these two aspects are the main differences between LDSA and the approach proposed by Nguyen et al. (2022). Thus, comparing LDSA's performance against these may provide important insights regarding which role assignment approach is optimal.\n\n**(Minor Weakness) Lack of diversity in environments used for evaluation.**\n\nWhile this only provides a minor improvement, the generality of this work's results can be further demonstrated by showing it works in a wider range of environments. For instance, Google Research Football and Overcooked are two environments where subtask assignment is required for learning.\n\n**Clarity:**\n\n**(Major Strength) The work is generally well written and very easy to understand.**\n\nOther minor suggestions to improve clarity will be provided in the questions/suggestions section below.\n\n**Significance:**\n\n**(Major Strength) The results of this work has potential significance to people focusing on dynamic task assignment in MARL.**\n \nThe idea of jointly learning subtask and ability representations can be an essential contribution if it is demonstrated to perform better compared to the baselines mentioned above. However, further experiments are required to demonstrate the significance of this work.\n\n**Citations:**\n\nWang et al. 2020. ROMA: Multi-Agent Reinforcement Learning with Emergent Roles. ICML 2020\n\nWang et al. 2021. RODE: learning roles to decompose multi−agent tasks. ICLR 2021.\n\nNguyen et al. 2022. Learning to Transfer Role Assignment Across Team Sizes. AAMAS 2022.\n\nLi et al. 2021. Celebrating Diversity in Shared Multi-Agent Reinforcement Learning. NeurIPS 2021.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "es4eypx2Cze", "reviewer": "Reviewer_VLj9", "summary": "This paper introduces an algorithm for efficiently training diverse abilities/policies in cooperative multi agent tasks. It learns to decompose a task into subtasks, match agents to a subtask based on their action-observation trajectory, and train policies for each subtask from the experience of agents assigned to it on that timestep. It does better than QMIX, ROMA and RODE on the harder tasks on the StarCraft II micromanagement tasks.", "questions": "- In the ablations section: Is there a way to decrease LDSA parameters and compare to the original-sized QMIX rather than increase QMIX parameters?\n", "limitations": "Limitations - they mentioned it may not improve things on simple tasks. I would also add that this is formulated only for a fully cooperative game at the moment.\nThe author didn't discuss potential negative social impacts, it would be good to mention even if the answer is inconclusive.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Well organized and clearly written paper.\n- Original and elegant method for decomposing subtasks without prior knowledge, and also without limiting the action space per subtask (like RODE) which is more flexible. \n- It does beat the other algorithms on harder tasks, which is a significant contribution.\n- The ablations, and other investigations (of e.g. effect of number of subtasks) helpful for understanding the different parts of the algorithm.\n\nWeaknesses:\n- Not sure why comparing only to QMIX style mixing networks makes for a fair comparison; why would comparing to other types of approaches not be fair?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "QosJ04L8BEQ", "reviewer": "Reviewer_7hXy", "summary": "Sharing parameters among all agents usually limits the behavioral diversity while learning a separate policy for each agent without any parameter sharing usually leads to high training complexity.\nTo balance the training complexity and the diversity of agents’ behaviors, this paper proposes a dynamic parameter sharing framework LDSA. LDSA divides the agents into up to $k$ groups (where $k$ is a manually set parameter) according to their action-observation histories. All agents share the parameters of the input layer and the GRU layer of the individual Q network. For the output layer, only the agents within the same group share the same parameters and the agents in different groups use different parameters. Experimental results in SMAC valid the benefits of the proposed dynamic parameter sharing mechanism.", "questions": "Please see the weaknesses above.", "limitations": "The group number $k$ has to be manually set aforehand for each task.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n  * The paper is well motivated and clearly written.\n  * The proposed framework LDSA is technically sound.\n\n**Weaknesses**\n  * (1) The proposed method has little to do with \"dynamic task assignment\". The core of LDSA is dividing the agents into up to $k$ groups based on their different behaviors and making the agents in different groups do not share parameters. There are no explicitly pre-defined tasks for the agents to solve. Therefore, using the words \"dynamic parameter sharing according to agents' behaviors\" may be more appropriate.\n  * (2) Important related works and baselines are missing. [1] and [2] propose different methods to balance the training complexity and the diversity of agents’ behaviors by dynamically sharing parameters among agents. I notice that the authors cited these two works in the paper but not discussed and compared with them. These two works should be well discussed in the related work and compared in the experiments.\n  * (3) The experimental evaluations in SMAC is not convincing. Recently, [3] and [4] have verified that the optimized QMIX (completely sharing the parameters among all agents) could achieve 100% win rates on all Easy, Hard and Super Hard scenarios of SMAC. Therefore, SMAC may not be a good testbed to validate the benefits of the dynamic parameter sharing mechanism. The authors could conduct more experiments in more complex environments, e.g., the challenging Google Research Football (GRF) [5].\n\n\n**Reference**\n  * [1] Christianos F, Papoudakis G, Rahman M A, et al. Scaling multi-agent reinforcement learning with selective parameter sharing[C], \n  * International Conference on Machine Learning. PMLR, 2021: 1989-1998. \n  * [2] Chenghao L, Wang T, Wu C, et al. Celebrating diversity in shared multi-agent reinforcement learning[J]. Advances in Neural Information Processing Systems, 2021, 34: 3991-4002.\n  * [3] Hu J, Jiang S, Harding S A, et al. Rethinking the implementation tricks and monotonicity constraint in cooperative multi-agent reinforcement learning[J]. arXiv e-prints, 2021: arXiv: 2102.03479.\n  * [4] Hao X, Wang W, Mao H, et al. API: Boosting Multi-Agent Reinforcement Learning via Agent-Permutation-Invariant Networks[J]. arXiv preprint arXiv:2203.05285, 2022.\n  * [5] Kurach K, Raichuk A, Stańczyk P, et al. Google research football: A novel reinforcement learning environment[C], Proceedings of the AAAI Conference on Artificial Intelligence. 2020, 34(04): 4501-4510.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "B6koww7rZ3u", "reviewer": "Reviewer_i6UD", "summary": "The paper proposes an algorithm to learn dynamic subtask assignment (LDSA) in cooperative MARL. First, it uses a subtask encoder to construct a vector representation for each subtask. Then, it proposes an ability-based subtask selection strategy to dynamically group agents into the similar subtask. In addition, the authors employ two tricks to make the representations of subtasks distinguishable and the learning process steady. Empirical results show that LDSA learns reasonable and effective subtask assignment for better collaboration and significantly improves the learning performance on the challenging StarCraft II micromanagement benchmark. ", "questions": "- We wonder how this method would perform if the multi-agent cooperative tasks do not require diverse agents' policies. For example, in Level-Based Foraging (GitHub - semitable/lb-foraging: Level-based Foraging (LBF): A multi-agent environment for RL), agents need to cooperate in strong coordination to collect foods with different levels where different types of policies are not necessary. We wonder if the effectiveness of LDSA depends on the type of problem. \n- We wonder why the performances of ROMA and RODE are considerably worse than those in their original paper. For example, RODE can achieve above 80% winning rate on the map of corridor and 6h\\_vs\\_8z, while in this paper, RODE cannot learn anything after 5 million timesteps. \n- Are the trajectory encoders shown in Figure 1(b) & 1(c) the same? \n- As the authors claim in the main text, the agents learn different abilities after training, so we wonder how these agents could perform tasks different from the training phase, which are known as ad-hoc tasks. \n", "limitations": "The authors do not claim the limitations of their method in the paper. We think, as mentioned above that the effectiveness of LDSA depends on the type of problem it aims to solve since the diversity of policies is not directly related to the effectiveness of cooperation. The authors should carry out more experiments on other different environments. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n- The writing of the paper is clear and structured. \n- Previous works dealing with multi-agent cooperative tasks predefine the task decomposition using prior domain knowledge, while this work does not use prior knowledge to decompose the task. \n- The number of subtasks should be set in advance but is a robust hyperparameter that all the scenarios in the SMAC benchmark can share an identical number of subtasks. \n- The ablation study demonstrates the effectiveness of the two regularizations and different components of LDSA. \n\nWeaknesses: \n- Since the number of subtasks $k$ can vary, the subtask defined in Definition 1 is more self-defined than objectively existing, making the definition informal. We think the authors should clarify the assumption under which the overall task can be decomposed into subtasks. \n- Ground truth about the subtasks is not available, so we prefer that this method works on the SMAC benchmark because it encourages the diversity of agents’ policies which encourages exploration. We think it would be better if the authors could clarify the assumption that the more diverse the policies, the better the agents would perform. \n- The experiments are only carried out on the SMAC benchmark, which restricts the reliability of the results. As far as we know, after fine-tuning, even a baseline like QMIX can achieve excellent performance on SMAC (GitHub - hijkzzz/pymarl2: Fine-tuned MARL algorithms on SMAC (100% win rates on most scenarios)). We believe it would make the results more convincing if the authors could conduct more experiments on other types of environments. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept