Decompose a Task into Generalizable Subtasks in Multi-Agent Reinforcement Learning
Authors: Zikang Tian
Ruizhi Chen
Xing Hu
Ling Li
Rui Zhang
Fan Wu
Shaohui Peng
Jiaming Guo
Zidong Du
Qi Guo
Yunji Chen
Venue: NeurIPS 2023 poster (2023)
Keywords: Multi-Agent Reinforcement Learning
Transfer Learning
Zero-Shot Generalization

Abstract: In recent years, Multi-Agent Reinforcement Learning (MARL) techniques have made significant strides in achieving high asymptotic performance in single task. However, there has been limited exploration of model transferability across tasks. Training a model from scratch for each task can be time-consuming and expensive, especially for large-scale Multi-Agent Systems. Therefore, it is crucial to develop methods for generalizing the model across tasks. Considering that there exist task-independent subtasks across MARL tasks, a model that can decompose such subtasks from the source task could generalize to target tasks. However, ensuring true task-independence of subtasks poses a challenge. In this paper, we propose to \textbf{d}ecompose a \textbf{t}ask in\textbf{to} a series of \textbf{g}eneralizable \textbf{s}ubtasks (DT2GS), a novel framework that addresses this challenge by utilizing a scalable subtask encoder and an adaptive subtask semantic module. We show that these components endow subtasks with two properties critical for task-independence: avoiding overfitting to the source task and maintaining consistent yet scalable semantics across tasks. Empirical results demonstrate that DT2GS possesses sound zero-shot generalization capability across tasks, exhibits sufficient transferability, and outperforms existing methods in both multi-task and single-task problems.

Decompose a Task into Generalizable Subtasks in Multi-Agent Reinforcement Learning
Zikang Tian 1.233 Ruizhi Chen', Xing Hul:5, Ling Li Rui Zhang', Fan 2u2.3.4, Shaohui Peng'. Jiaming Guo Zidong Du Qi Guo Yunji Chen 1.2
SKL of Processors, Institute of Computing Technology, CAS, Beijing, China University of Chinese Academy of Sciences, Beijing, China $Cambricon Technologies, Beijing, China
Intelligent Software Research Center, Institute of Software, CAS, Beijing, China Shanghai Innovation Center for Processor Technologies, SHIC, Shanghai, China {ti ftianzikang21s, cyj syj]@ict.ac.c
Abstract
In recent years, Multi-Agent Reinforcement Learning (MARL) techniques have made significant strides in achieving high asymptotic performance in single task. However, there has been limited exploration of model transferability across tasks. Training a model from scratch for each task can be consuminn and expensive, especially for large-scale Multi-Agent Systems. Therefore, it is crucial to develop methods for generalizing the model across tasks. Considering that there exist task-independent subtasks across MARL tasks, model that decompose such subtasks from the source task could generalize to target tasks. However, ensuring true dask-independence of subtasks poses challenge. In this paper, we propose to decompose a task into a series of generalizable subtasks (DT2GS), a novel framework that addresses this challenge by utilizing scalable subtask encoder and an adaptive subtask semantic module. We show that these components endow subtasks with two properties critical for boss-independence: avoiding overfitting to the source task and maintaining consistent yet scalable semantics across tasks. Empirical results demonstrate that DT2GS possesses sound zero-shot generalization capability across tasks, exhibits sufficient transferability, and outperforms existing methods in both multi-task and single-task problems.
Introduction
In the last few years, many works in MARL field prospers, including value-based algorithms
[26] [25] 20, 27
30
39
TO 17). and policy-based algorithms [6, 36 335 [1] 24 (7]. However,
these works mainly focused on the model's asymptotic performance in a single task, neglecting its transferability across tasks. Training an optimal model on a single task requires millions of
interactions with the environment 21 [6], particularly when dealing with large-scale Multi-Agent Systems, whereas transferring the model across tasks can reduce training cost by dozens of times 5.. As the number of tasks increases, the reduction in training costs due to model transfer will become even more significant.
Knowledge reuse is common approach for model generalization across tasks in the MARL field [4]. Recent knowledge reuse methods for online MARL can be roughly classified into two categories: network-design- based methods and task-embedding. based methods. Network- desin---einn--eeee based methods
33 9 implicitly reuse knowledge extracted from the source task to the target task by constructing
"Corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

universal model structure across tasks by utilizing Population Invariant Structure such as Transformer [ or GNN [ However, it's unclear whether the knowledge extracted from the source task is suitable for the target task. bark---mbedding-bbaeeeeee
methods [19 23 14 reuse knowledge by calculating task similarity using learned task embeddings that capture task dynamics. However, accurately mapping task dynamics to task embeddings requires numerous tasks as samples. In word, current knowledge reuse methods for online MARL still have limitations, that is, inefficient knowledge reuse and reliance on a large number of tasks samples. Additionally, we provide a section for related work in Appendix to introduce these methods more specifically.
Compared with the common knowledge reused in these methods above, there exists an alternative knowledge named bask-independent subtasks that can relieve these two limitations. These task- independent subtasks, such as "hit and run". "focuse fire", "disperse and line up". etc, are certainly applicable across tasks [41], improving the efficiency of knowledge reuse. Besides, these task- independent subtasks can be decomposed from few tasks, so as to remove the reliance on large number of tasks samples. However, ensuring the task-independence of decomposed subtasks is a challenge. Task-independennee of subtasks refers to their effectiveness across tasks, which requires two essential properties: (I) avoiding overfitting to the source task, (2) maintaining consistent yet scalable semantics across tasks. In this paper, propose DT2GS, novel framework devoted to generalize model across tasks by decomposing subtasks from the source tasks and endowing them with these two properties necessary for task independence.
The proposed framework, DT2GS, aims to decompose a task into series of generalizable subtasks, as shown in Figure [ DT2GS is primarily composed of two parts: the scalable subtask encoder and the adaptive action decoder. The scalable subtask encoder assigns a subtask to each agent based on its (subtask, entity-observation) history rather than the (action, observation] } history. This approach helps prevent the process of assigning subtasks from overfitting to the source task. The adaptive action decoder leverages the assigned subtasks and current entity-observations to calculate the specific actions for interacting with environment. Within this decoder, the adaptive subtask semantic module ensures that the assigned subtasks have consistent yet scalable semantics across tasks based on their effects on entities. Based on the experimental results, our framework exhibits several desirable properties follows:
zero-shot generalization capability: the trained model can be effectively depolyed to multiple target tasks without any fine-tuning;
robust transferability: significantly accelerated model convergence on complex target
tasks and achieving average speedup of 100x;
better asymptotic performance: achieved state-of-the-art (SOTA) results on multi-task
and single-task problems;
better subtask interpretability: decomposed lask-independent subtasks with practical behavioral semantics that consistent yet scalable across tasks
2 Preliminary
2.1 Background
In this paper, fully cooperative multi-agent system (MAS) is described decentralized partially observable Markov decision processes (Dec-POMDP) [88, which is defined by a tuple G =< S,A, O, 7. n, A, Q2 R, >. S is the global state space. A is the action space shared for all agents. O is the shared individual observation space for all agents. 7 is the discount factor. n is the number of agents. A an individual observation of E O from dynamic environment according to the observation function ((s :). And if we suppose agent are controlled by policy म, which takes of or history individual observations Tf as input and parameterized by 6,, agent i will select an action at according to x (aa) lof). Therefore, a joint action A' (a1, .2,..., an) E A will be formed where a, corresponds to agent i. After passing the joint action A' into the environment, global reward signal r, R(s', A²) shared by all agents will be received and the environment will transmit into next state s1+1 according to the transition function P(s (s', A') S x Ax S
...,, m1] is the set of agents our algorithm controlled. At each timestep f, agent i obtains
[0, 1]. The goal of agents is defined as finding the

---

policy т to maximize the objective function
2 ((x) = E 7 R(s (
(I)
where 7" R(s A is the discounted return of all agents.
2.2 Generalizable model structure in MARL
For model generalization in MAS, a generalizable model structure is necessary for addressing the problem of varying state/observation/actti space (S/O/A) across tasks. Here define the agents controlled by MARL policy and agents built-in tasks as entities. In this paper, we use n, m. Rally, Renemy denotes the number of agents, entities, allies and enemies, respectively. And the following equation holds: n Mailyy + 1, m = + Renemy- As demonstrated in ASN (33], concatenation of m entity-observations: oi where o,,1 is the observation of agent on itself and environment, and the rest are the observations of agent other 1 entities. Additionally, action space can be decomposed into two components: A which consists of actions affecting the agent itself or the environment, and A interaactvve which contains actions that directly interact with other entities. This aligmment between entity-observation and actions, which is referred as action semantics, forms the foundation for computing the value or probability of action based on its aligning entity- totey--obsevvttio,,, leading in generalizable model structure actoss tasks.
agent's observation
be constructed
[o..1. o........ m. m,,
Adaptive Action Decoder
FC äu Gumbel Saltmax
A
Meat-
Decision
Cognilion
a
Adaytive Subnask Semantics
aity
onemy
Similarity
it Openator: Geunsion Product
Emtedding
Embreddin
onily
onewy
emt()
Coprition
Adaptive Subteak Semrantcs
Scataste Subtack Encader
Scatable Subtack Encader
emb(oniv)
emb(o
emb(k)
O Ry
On Kay
Figure 1: DT2GS Framework. The DT2GS Framework comprises two modules: the Scalable Subtask Encoder and the Adaptive Action Decoder, where the Adaptive Subtask Semantic module serves as the core of the Adaptive Action Decoder. The Scalable Subtask Encoder effectively assigns subtasks agents without overfitting the source task. With the Adaptive Subtask Semantic module endowing assigned subtasks with consistent yet scalable semantics across tasks, the action decoder takes the ontity-observations and subtasks as inputs to generate actions for interacting with the environment.
3 DT2GS Framework
DT2GS primarily consists of two components: the scalable subtask encoder and the adaptive action decoder, as shown in Figure DT2GS begins with scalable subtask encoder, which assigns subtask to each agent based on its (subtask, entity-observation) history instead of the (action, observation] history, so as avoid overfitting to the source tasks. Then the adaptive action decoder calculates the specific actions for interacting with environment by leveraging the assigned subtasks

---

and current entity-observan plays a core role, ensures that the assigned subtasks have consistent yet scalable semantics across tasks based on their effects on entities. With the proposed scalable subtask encoder and the adaptive subtask semantic module, DT2GS endows the decomposed subtasks with task-independence, leading model generalizable across tasks. A pseudocode is provided in Appendix H to illustrate the DT2GS framework.
Within this decoder, the adaptive subtask semantic module, which
c-1
kf
hff
k
h
c++1
Scalable Subtask Encoder
of
af
task deskndependent
kE
fof
k
RC
C+1
musk--independent
Figure 2: Scalable Subtask Encoder. The Scalable Subtask Encoder assigns subtasks to each agent based on its behavioral history representation, which is depicted by its [subtask, entity-observation) history instead of commonly used [action, observation] history. The red dashed-line box denotes the ( (action, observation) history, which is dask-dependent. Specifically, at each timestep t, agent obtains its subtask K by utilizing its observation of, last action a history latent embedding hi-1 produced by RNN. On the contrary, the red solid-line box denotes the (subtask, ontity-observation history, which is task-independent. Specifically, at each timestep ++1
agent i obtains its subtask {o } [o 0 tf J. last subtask kt history latent embedding hy produced by RNN.
k kt+1 by utilizing its
3.1 Scalable Subtask Encoder
agent's history, which comprises its observations and behavioral history, contains valuable information about its characteristics, such as velocity, attack distance, etc. Therefore, analyzing an agent's history can help us identify the most suitable subtask for the agent to perform. Typically [38) recurrent neural network, such LSTM [8] or GRU (3), is employed to construct the agent's history representation from its (action, observation history. The history representation is then passed to softmax operator, which samples a one-hot subtask for the agent. However, since actions and observations are desk-depen,ent, this approach may lead to subtask encoder that overfits to the source task, making the source model non-generalizable. Considering that the subtasks history of an agent not only reflects its behavior history, but also is task-independent, and the onttry-observationss history more general across tasks than observations history, our scalable subtask encoder assigns subtask to each agent based on its (subtask, -ntity-observation history rather than (action, observation) history, as illustrated in the red solid-line box in Figure
The detail the scalable subtask encoder demonstrated in Figure I right. By replacing ob- servations with entity-observations,, we first design a structure that can obtain fixed-dimensional observation embeddings, regardless the number of entity-observation Additionally, since permuting the order of entity-bservvttions in the observation does not change the information!! [13], we use a Permutation Invariant operator, namely, Gaussian Product, to obtain the observation embedding. Specifically, we first MLP parameterized by 0, to embed the eniity-observation as an entity-embedding:
contained in the observation.
(pi.)) fo. (o.g)) 1,..,
(2)
and the observation embedding c/, which also referred to as Env Cognition of agent in Figure is constructed as:

---

of ~NN(L where N N of & II N (u
(3)
Then utilize trajectory encoder based GRU, which is parameterized by 0h, to obtain an agent's history representation. The GRU takes an agent's observation embedding ef, last subtask 4 and previous hidden state n 11 as inputs, and generates a new hidden state hf as its history representation:
h = fer (ef K (e
Afterwards, we use the Gumbel-Softmin trick with categorical reparameterization 12 subtask b to agent based on its current history representation h;:
to assign
A (ofmbal-Softmaxhh))
where k E R is n4 nu-diimesioonal one-hot vector and n& is hyperparameter denoting the total number of subtasks. The Gumbel-Softma operator allows our process of subtask assignment trainable by the way of gradient backpropagation.
self
galyy
alls'natty
cRemy:
enemytyie
Zg
Wy
W
We
Matmul Scale Sof Softmax
aally
Matmul
enemy
Aasee
8 Jla8
altynally
gpenmmy's
enemo'ee
oenemy
k
Qvatue Pr
Figure 3: Adaptive Subtask Semantic module. The Adaptive Subtask Semantic Module en- dows subtasks and actions with adaptive semantics. denotes the embedding of subtask k.
Entity-oseevvttions are denoted by oo gallys ..... o aallymatiy onemy. ..... O emernyenemms (we replace Teeney with ne in Figure 3 for convenience). With subtask embedding and entity-
observations, obtain the semantics du of subtask k and the semantics of actions
so gallys allynaily enemyn eenemynnnmy by utilizing the Attention mechanism [29]
3.2 Adaptive Subtask Semantics
The generalizable subtasks should maintain consistent yet scalable semantics across tasks. The traditional method of constructing subtask semantics is embedding one-hot subtasks into vector representations based MLP, However, this approach endows target subtasks with completely identical semantics source subtasks when deploying the source model to the target tasks, neglecting the difference between source and target tasks so to restrict the zero-shot generalization of subtask semantics across tasks. To overcome this limitation, our adaptive subtask semantic module leverages inductive bias that subtask semanties refers to the effects of an agent on entities when performs given subtask. This approach endows subtasks with semantics that can adaptively adjust based on the agent's 's effects on entities, thereby enabling greater zero-shot generalization of subtask semantics across tasks.
5

---

Since a subtask's 's semantics actually refers to the effects of an agent on entities when it performs this subtask, we construct subtask semantics as the weighted sum of entities, where the weight represents the degree of effects. And considering that we have obtained one-hot subtask and entity-osservation, we utilize the Attention mechanism [29] without position embedding, which is scalable across tasks regardless of the number of entities and permutation invariant about the order of ooseritobservaitonss in the observation, to obtain the subtask semantics, as illustrated in Figure Specifically, we take one-hot subtask embedding
= (mbeddinggA:))
(6)
as the query and intity-bbservation embedding as the keys well as values: Q! Wa K W V = Woo where Wo, WX, Wv are learnable parameters and of = [o 1. o o,,l
Then we construct adaptive subtask semantics of as follows:
Q.KT" vdx
= softmax
V
=
(
where dx the feature dimension of KJ. Additionally, to make the process of obtaining actions scalable across tasks, we extended the model structure of ASN (33). Since the alignment between actions and entity-observations may discrepant from the prior, we construct the adaptive action semantics of [sf. 10f 29,::.. ... of by utilizing the Self-Attention mechanism taking the entity- observations as inputs:
O.K!T
softmax
Vr,
Q Q ==
88)
vdk
Subsequently, we calculate the value or probability of each action in current observation by comparing the similarity of subtask semantics with action semanties:
Quatue (ay (o)) or Pr(aylo() = similarity(@ pp.s))
(9)
where similarity is trainable MLP taking the concatenate of of and fh.5 as inputs.
4 Experiments
4.1
Experimental Setup
We evaluated the performance of DT2GS on the StarCraft Multi-Agent Challenge (SMAC) 22 and the multi-agent particle world environments (MPE) [16] (shown in Appendix D) SMAC contains several tasks that are similar but different, such as the marine-series tasks (13m, 8m, 8m_ imm, 10m_vs_ 1mm 25m, 37m_vs_30m) and the seaker_zeallt-series tasks (2s3z, 3s5z, 3s57_vs_3s62) Besides, changing the number of agents and landmarks in MPE can also form a series of tasks. These tasks met our requirements of cross-task generalization, allowing us evaluate the ability of DT2GS to generalize across different tasks.
In the generalization capability part of experiments, selected ASN 33 and UPDeT 9 baselines. ASN was chosen because it promotes the development of universal models across tasks MARL. For the sake of fairness in comparison, we make ASN generalizable across tasks by utilizing the attention mechanism and use " "SSN G" to denote this generalizable ASN. UPDeT was selected because constructs universal model across tasks via policy decoupling with self-attention, which is also used in the adaptive subtask semantic module of DT2GS in Sec 3.2 In addition, we implemented DT2GS, UPDeT, and ASN_G G based on MAPPO 40 which is considered SOTA in on-policy MARI.
In the asymptotic performance part of experiments, we added another five basclines for comparison: 40]. LDSA (38), ROMA (1), RODE (32, and HSD [3). LDSA, ROMA, RODE, and HSD focus on concepts like skills/options/roles in MARL, which are similar to subtasks studied in method.
Additionally, We the number of subtasks n& to 4 and averaged all results over random seeds. And the experiments are arranged as follows: Firstly, we evaluated the zero-shot generalization capability of DT2GS across tasks. Secondly, we analyzed the practical semantics of subtasks on the marine-series tasks. Thirdly, we conducted several experiments to exhibit the transferability of DT2GS across tasks. Finally, we demonstrated that DT2GS achieves SOTA performance on multi-task and most single-task problems in terms of asymptotic performance.

---

4.2 Zero-Shot Generalization across Tasks
In this section, we designed 8 different zero-shot generalization scenarios, where each scenario includes target task that is more difficult than the source task. That is say, the relationship from the source task to the target task is extrapolated. For example, the target task may be larger in scale (like 8m 25m) contain greater disparity in military strength than the source task $3s_vs_47 -> 3s_vs______ $0_vs_50 vs_5z). We deployed the model trained on the source tasks to the target tasks without any finetune. As shown in Figure 4 DT2GS significantly outperforms UPDeT and ASN G in terms of zero-shot generalization capability, achieving an average test winning rate surpass of about 22% and 34%, respectively, over all 8 zero-shot generalization scenarios.
Zero-Shot
algo DT2GS
0.8
0.2
UPDeT
6448
0.009
ASN_
eval win rrate
0.439
0.484
0.4
0.336
0.2
0:203
0,208
43 0.88
64
0,
022
(b)
(c)
(d)
(f)
(g)
(h)
The figure shows a comparison of zero-shot generalization capability between DT2GS, UPDeT, and ASN_G across various source and target tasks. The horizontal axis represents the source task - the target task, where (a) 33_vs_42 + 3s__s_______ vs 5z, (b) 2s3z + 3s5z, (c) 3s5z - 3s5z_vs__ vs________ $5$__vs_366,, (d) 8m - 8m_ vs_9m, (e) 8m + 10m_vs vs 1m. (f) 8m - 25m, (g) 8m_ ____v_______ 1s_9mmm_-____mm + 25m, and (h) 8m vs 9mm - 5mm_ vs 6m. The vertical axis represents the winning rate when deploying the source model to target task without any finetue. The red, green, and purple histograms correspond to DT2GS, UPDeT, and ASN G, respectively. The missing histograms indicate that the winning rate of deploying the source model target task is 0%.
Figure 4:
4.3 Analysis of Subtask Semantics
In this section, we analyzed the practical semantics of subtasks based on the marine-series tasks. We first trained a DT2GS model from scratch on the 8m task and then deployed it to 8m and 10m_ws vs sa__v__lin 1mm, as shown in Figure 5 left. In the 8m and 10m vs_ 1lm tasks, the DT2GS agents initially utilized subtask_ to scatter the formation and advance towards the enemy at t = 1. When the enemies 6, most of these agents selected subtask 2 to focus fire on enemies while also paying attention to their own health in order to move back when in danger, thereby avoiding the decrease firepower caused by personnel reduction. And other agents chose subtask_4 4 to slightly adjust their relative position with the enemies to seek better position for firepower output. At 13, some agents sensed that their allies' health was too low while their own health was safe enough. These agents switched subtask from subtask 2 to subtask_3 3, which means charging. And the specific behavior was to attack while moving closer to the enemies' position to attract hatred as to avoid the decrease in firepower caused by the death of allies. At the end of the battle (after f 21), all agents changed their subtasks to subtask_3 to concentrate their firepower to end the battle. We observed that under the subtask of charge, even if an agent was close to death, would not move back to disperse hatred as overall withdrawal would lead to failed combat.
reached the attack range of DT2GS agents
Furthermore, we conducted zero-shot generalization experiments on additional marine tasks, and the corresponding change process of subtask percentage shown in Figure We observed that the change process of each subtask's percentage in the source task (8m) is quite similar to that in the target tasks (10m_ _a_vs__ vs_1______ vm_ 5m_vs vs_ s5_6,, 8m vs_ vs_9m, 25m, and 27m_ sip__s__3000 (s_30m. This result demonstrates that the subtasks decomposed by DT2GS from the source task (8m) are iask-independent across tasks and
7

---

disperse hit and run charge
8m
pos adjust
13
10m
11m t=13
t=1
t=6
t=7
t=21
t=24
Subtask Percentage
Figure 5: Visualization of the practical semantics of subtasks (left) and the change process of subtask percentage (right) in 8m (up) and 10m_y vs Im (down). The results for both tasks are obtained from the source model's evaluation on the corresponding task, where the source model is acquired by learning from scratch in task 8m.
therefore effective for the target tasks. Both results from Figure 5lnd 6 suggest that DT2GS policy has sound zero-shot generalization capability by decomposing the task into series of lask-independent subtasks.
subtask's percentage when 8in'ss _sot
Bm Ownn]
5m, vr, (win)
Bm,, vs. (win))
1.0
1.0
vubtank peroent
0.8
sibtak pertett
0.8
0.8
chsparaa
dispern
daperse Itl und run charge
1il and run
lsl and run
change
chaine
pos ad usz
pos aljust
pos acjist
0.2
0.2
0.2
Uimestepp
Limestep
10m 1 (win)
75m (win)
27m_ys Vs, 30m [ose)
1.0
1.0
1.0
vubtan percent
0.8
subtak percet
0.6
0.8
chipnn
dispamse
disperse hi anc run
tit and run
tel. and run
0.4
charge
pos ad usi
pos aQjust
charge
0.4
charge pos acjust
0.2
0.2
Figure 6: Changes of subtask percentage when deploying the source model, which is learned from
scratch on 8m, to marine tasks. The subtask percentage caleulated as p(k) = N. the percentage subtask k. N₃ the number of agents that select subtask k, and N, is the total number of agents.
No. where p(k)
4.4 Transferability across Tasks
Figure 7 (a) illustrates four scenarios that we designed to evaluate the transferability of DT2GS policy. In all four scenarios, the relationships from the source task to the target task are extrapolation, including increase in entity types (3s5z 3s6z + Ic3s5z), an increase the number of enemies (3s5z + 3s52_vs 33662 and $s_vs_4z + 38 vs. 52), as well as an increase in the total scale of entities (8m_vs_9) vm + 27m_ 33_m0m).
As we can see, DT2GS finetune demonstrated significantly efficient convergence as well as higher and more stable asymptotic performance compared with UPDeT finetune and ASN G finetune in all transfer scenarios. Notably, in the scenario where source task was 385z (easy) and the target task was 3557_w vs_ v_3s6z (superhard), DT2GS Sinnetufintueeeeee achieved optimal performance with only 6400 steps interaction with environment, whereas learning from scratch required at least million steps. Furthermore, compared to other baselines of learning from scratch, DT2GS finetune accelerated

---

the model's convergence on target tasks by an average of x00x. In conclusion, DT2GS finetune
exhibited better performance than baselines on the following evaluation metrics 28 4 jumpstart. time to threshold and asymptotic performance, demonstrating significantly sufficient transferabilityy
Fnolune
Single Tock
vbd 24
ACC Ao
3
15
(a) Transferability
(b) Single-Task
Figure 7: (a) Comparison of transferability between DT2GS and baselines, where baselines include transfer of UPDeT (UPDeT finetune), ASN G (ASN G finetune) and learning from scratch of DT2GS, UPDeT, ASN, MAPPO. (b) Comparison of performance on Single-Tare Task between DT2GS and baselines, including UPDeTT, ASN, MAPPO, HSD, RODE, ROMA and LDSA, on 3 superhard tasks (3s57_ vs _s_3,, 6h_ vs 8z, corridor) and hard task (5m_ vs_ 3mm_v_66m)
4.5 Performance on Multi-task and Single-task
In this section, we first designed 2 multi-task problems, including the marine-series tasks ((3m, 8m, 8m_vs_ mmm_ 10m_vs_ demonstrate the representational capacity of DT2GS. In each multi-task problem, the policy interacted synchronously with multiple tasks that make up this multi-task to collect data, which was shuffled and used to update the policy. As shown in Figure 88] DT2GS exhibited better learning efficiency and more stable asymptotic performance compared with UPDeT and ASN_G.
vs "ss_1mm 11m)) and ciakerrzeelo--eeiiee tasks ([2s3z, 3s5z, 3s5z_ vs_ vs_3s6z]), to
1.20
1.00
1.03
0.99
0:50
0.35
0.35
0.28
0.25
0.4
J0075
slaga
248
1.00
3.00
1.03
0.50
0:5
0.33
0.35
0.25
0.33
AMM
ASS
"Rny
stage
110m 1oor)
1.30
1.00
1.43
0.75
0.50
072055
0.53
0T2O5
0.25
0.25
mepp
D07
stepe
367
mtaps
Figure 8: Comparison of performance on multi-task problems between DT2GS and baselines, including UPDeT and ASN_G. G.
Subsequently, we also compared DT2GS with baselines on single-tasi scenarios. We selected four representative scenarios, including superhard/ / hard tasks, as shown (bg while performance on other scenarios is presented in the Appendix B Our evaluation metrics included learning efficiency and final asymptotic performance. Compared to MAPPO, DT2GS significantly improved both metrics, particularly in superhard tasks such as 3s52_ vs_________ f3_3s62 and 6h 6h_vs___ 8z. In addition, compared to other baselines based on MAPPO, including UPDeT and ASN, DT2GS consistently outperformed them in terms of learning efficiency and asymptotic performance, on average.

---

Conclusion
Model generalization has emerged as a promising approach to reduce training costs. In this paper, we proposed DT2GS, an effective approach for addressing the problem of model generalization across tasks in the MARL field. Our insight is that aask-independent subtasks exist across tasks, making it possible to generalize the model across tasks. Based on this insight, assumed that the model can be generalized to target tasks if it can decompose indeeendnnndn subtasks from source tasks. The challenge then becomes ensuring that the subtasks we decompose from the source task are truly task-independent, Regarding this issue, we proposed two properties that enable task-independence of subtasks: (1) avoiding overfitting to the source task, (2) maintaining consistent yet scalable semantics across tasks. Then we proposed DT2GS to endow the subtasks with these two properties by introducing the scalable subtask encoder and the adaptive subtask semantic module, respectively, Empirical results demonstrated that DT2GS can decompose tasks into a series of generalizable subtasks, leading generalizable MARL policy. Nevertheless, it would be beneficial consider task-specific subtasks as well when there is significant distribution shift between source and target tasks. In our future work, we will focus on expanding the generalization scenarios of DT2GS to address this limitation.
6 Acknowledgements
This work is partially supported by the NSF of China(under Grants 61925208, U22A2028, 62222214, 62002338, 62102399, U19B2019, 92364202). CAS Project for Young Scientists in Basic Rescarch(YSBR-029)) Youth Innovation Promotion Association CAS and Xplore Prize.
References
[1] Akshat Agarwal, Sumit Kumar, and Katia Sycara. Learning transferable cooperative behavior in multi-agent teams. arXiv preprini arXiv:1906.0122 2019.
[2] Georgios Boutsioukis, Ioannis Partalas, and loannis Vlahavas. Transfer learning in multi-agent reinforcement learning domains. In Recent Advances in Reinforcement Learning: 9th European Workshop, EWRL 2011, Athens. Greece, September 9-11, 2011, Revised Selected Papers 9, pages 249-260 Springer, 2012.
[3] Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rmn encoder- decoder for statistical machine translation. arXiv preprint 14X.v:1406.1177, 2014.
[4] Felipe Leno Da Silva and Anna Helena Reali Costa. survey on transfer learning for multiagent reinforcement learning systems. Journal of Artificial Intelligence Research, 64:645-703, 2019.
[5] Sabre Didi and Geoff Nitschke. Multi-agent behavior-based policy transfer. In Applications of Evolutionary Computation: 19th European Conference, EveeAplicattions 2016, Porto, Portugal, March 30-April 1, 2016, Proceedings, Part II 19, pages 181-197. Springer, 2016.
[6] Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
[7] Qingxu Fu, Tenghai Qiu, Jianqiang Yi, Zhiqiang Pu, and Shiguang Wu. Concentration network for reinforcement learning of large-scale multi-agent systems. arXiv preprint arXiv:2203.064JJ 2022.
[8] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 983) 1997.
[9] Siyi Hu, Fengda Zhu, Xiaojun Chang, and Xiaodan Liang. Updet: Universal multi-agent rein- forcement learning via policy decoupling with transformers. arXiv preprint arXiv:2 100.0001,, 2021.
10

---

[10] Shariq lqbal, Christian A Schroeder De Witt, Bei Peng, Wendelin Böhmer, Shimon Whiteson, and Fei Sha. Randomized entity-wise factorization for multi-agent reinforeement learning, In International Conference on Machine Learning, pages 4596-4606. PMLR, 2021.
[11] Shariq Iqbal and Fei Sha. Actor-attention-ee International conference machine learning, pages 2961-2970. PMIRR 2019.
for multi-agent reinforcement learning. In
[12] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmast arXiv preprint arXiv: 1611.01144, 2016.
[13] HAO Jianye, Xiaotian Hao, Hangyu Mao, Weixun Wang, Yaodong Yang, Dong Li. Yan Zheng, and Zhen Wang. Boosting multiagent reinforcement learning via permutation invariant and permutation equivariant networks. In The Eleventh International Conference Learning Representations.
[14] Yong Liu, Yujing Hu, Yang Gao, Yingfeng Chen, and Changjie Fan. Value function transfer for deep multi-agent reinforcement learning based on n-step returns. In LCCA, pages 457-463. Macao, 2019.
[15] Qian Long, Zihan Zhou, Abhibav Gupta, Fei Fang. Yi Wu, and Xiaolong Wang. Evolution- ary population curriculum for scaling multi-agent reinforcement learning. arXiv preprint arXiv:2003.104233 2020.
[16] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb. OpenAl Pieter Abbeel, and Igor Mordatch, Multi-agent actor-critic for mixed cooperative-compettivy environments. Advances in neural information processing systems, 30, 2017.
[17] Shuang Luo, Yinchuan Li, Jiahui Li, Kun Kuang, Furui Liu, Yunfeng Shao, and Chao Wu, S2rl: Do we really need to perceive all states in deep multi-agent reinforcement learning? arXiv preprint arXiv:2206. 1054, 2022.
[18] Frans A Oliehock and Christopher Amato. concise introduction to decentralized POMDPs. Springer, 2016.
[19] Rongjun Qin, Feng Chen, Tonghan Wang, Lei Yuan, Xiaoran Wu, Zongzhang Zhang, Chongjie Zhang, and Yang Yu. Multi-agent policy transfer via task relationship modeling. arXiv preprint arXiv:2203.04433 2022.
[20] Tabish Rashid, Gregory Farquhar, Bei Peng, and Shimon Whiteson. Weighted qmix: Expanding monotonic value function factorisation for deep multi-agent reinforcement learning. Advances in neural information processing systems, 13:10199-10210. 2020.
[21] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. In International conference machine learning, pages 4295-4304 PMLR, 2018
[22] Mikayel Samvelyan, Tabish Rashid, Christian Schroeder De Witt, Gregory Farquhar, Nan- tas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr, Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. arXiv preprint arXiv: 1r.Xiv:1902.000 990200403, 2019.
[23] Lukas Schäfer, Filippos Christianos, Amos Storkey, and Stefano V Albrecht. Learning task embeddings for teamwork adaptation in multi-agent reinforcement learning. arXiv preprint arXiv:2207.02244 2022.
[24] Wenling Shang, Lasse Espeholt, Anton Raichuk, and Tim Salimans. Agent-centric representa- tions for multi-agent reinforcement learning. arXiv preprint wxXiv:2104.09402, 2021.
[25] Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In International conference machine learning, pages 5887-5896. PMLR, 2019.
11

---

[26] Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojeiech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Mare Lanctot, Nicolas Sonnerat, Joel Leibo, Karl Tuyls, et al. Value- decomposition networks for cooperative multi-agent learning. arXiv preprint arXiv: 1006170665522 0526,, 2017.
[27] Ming Tan. Multi-agent reinforcement learning: Independent vs. cooperative agents. In Proceed- ings of the tenth international conference on machine learning, pages 330-337, 1993.
[28] Matthew E Taylor and Peter Stone. Transfer learning for reinforcement learning domains: A survey, Journal of Machine Learning Research, 10(7), 2009.
[29] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
[30] Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. Qplex: Duplex dueling multi-agent q-learning. arXiv preprint arXiv:2008. 01062, 2020.
[31] Tonghan Wang, Heng Dong, Victor Lesser, and Chongjie Zhang. Roma: Multi-agent reinforce- ment learning with emergent roles. arXiv preprint arXiv:2003.08033 2020.
[32] Tonghan Wang, Tarun Gupta, Anuj Mahajan, Bei Peng. Shimon Whiteson, and Chongjie Zhang. Rode: Learning roles to decompose multi-agent tasks. arXiv preprint arXiv:2010.01523 2020.
[33] Weixun Wang, Tianpei Yang, Yong Liu, Jianye Hao, Xiaotian Hao, Yujing Hu, Yingfeng Chen, Changjie Fan, and Yang Gao. Action semantics network: Considering the effects of actions in multiagent systems. arXiv preprint arXiv: 1907:1000111110 2019.
[34] Weixun Wang. Tianpei Yang, Yong Liu, Jianye Hao, Xiaotian Hao, Yujing Hu, Yingfeng Chen, Changjie Fan, and Yang Gao. From few to more: Large-scale dynamic multiagent curriculum learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 7293-7300, 2020.
[35] Yihan Wang, Beining Han, Tonghan Wang, Heng Dong, and Chongjie Zhang. Dop: Off- policy multi-agent decomposed policy gradients. In International Conference Learning Representations, 2020.
[36] Ying Wen, Yaodong Yang, Rui Luo, Jun Wang, and Wei Pan. Probabilistic recursive reasoning for multi-agent reinforcement learning. arXiv preprint arXiv: 09207, 2019.
[37] Jiachen Yang, Igor Borovikov, and Hongyuan Zha. Hicrarchical cooperative multi-agent reinforcement learning with skill discovery. arXiv preprint arXiv: 19/2.03588, 2019.
[38] Mingyu Yang, Jian Zhao, Xunhan Hu, Wengang Zhou, Jiangcheng Zhu, and Houqiang Li. Ldsa: Learning dynamic subtask assignment in cooperative multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 35:1698-177 1710, 2022.
[39] Yaodong Yang, Jianye Hao, Ben Liao, Kun Shao, Guangyong Chen, Wulong Liu, and Hongyao Tang, Qatten: A general framework for cooperative multiagent reinforcement learning. arXiv preprint arXiv:2002. 203999999...9 2020.
[40] Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo in cooperative multi-agent games. Advances in Neural Information Processing Systems, 35:24611 1 -24424, 2022.
[41] Fuxiang Zhang, Chengxing Jia, Yi-Chen Li, Lei Yuan, Yang Yu, and Zongzhang Zhang. Discovering generalizable multi-agent coordination skills from multi-task offline data. In The Eleventh International Conference on Learning Representations, 2023.
12

---

A Related Work
Transfer learning across tasks in online MARL Transfer learning across tasks is a promising approach to accelerate the convergence of models, especially in large-scale MAS. The first method of transfer learning in online MARL domains BITER (2]. which uses the joint policy learned in the source task to bias the initial policy of agents in the target tasks. After more than decade of development, some methods have been proposed in this field. In this regard, we classify these methods into two categories based their characteristics, namely dewwork-deiign-bssee methods and Task-embedding-baal methods.
The betwork-design-baseee methods implicitly reuse knowledge from source tasks by constructing universal model structure across tasks using Population Invariant Structure such as Transformer [9] or GNN [T]. Since the existence of varying state/observation/actio space across tasks, the network-design design-based methods are the foundation of all other methods. And the Task- emsed-nned based methods are knowledge transfer by calculating task similarity using learned task embeddings that capture task dynamics (2 5
14 23 19I. For example, MATTAR [9] models the target task representation weighted sum of series of source task representations and decodes the task representation to the policy weight that suitable for the task by utilizing a hypernetwork. And MATE [23] models task representation by reconstructing the task dynamics, so as achieve task differentiation and fast adaptation of policy on tasks.
Previous works focused on multi-agent transfer learning based network-design or task-embedding, but lacked of efficient knowledge reuse, leading to limited generalization capability. Our work (DT2GS) leverages knowledge reuse by ensuring semantic consistency and scalability between diverse tasks and preventing over-fitting of source tasks, which greatly improves transferability and even zero-shot generalization capability.
Transfer learning across tasks in other paradigms There are some methods dedicated to transfer learning across tasks in MARL belonging to other paradigms. For example, ODIS [41], belonging
to the offline RL paradigm, discovers generalizable skills from multi-task offline data to enhance the model's generalization capability to unseen tasks. Besides, EPC [15 and DyMA-CL knowledge transfer across tasks by way of Curriculum Learning.
34 fulfill
htlll//pptioon/ooe//uuttsass in MARL There are some works in MARL focusing on concepts like skills/options/roles, which are similar subtasks studied in our method. LDSA [3 learns dynamic subtask assignments and constructs different policies for different subtasks, so enabling agents to possess specific abilities to handle different subtasks. ROMA 31 introduces the role concept into MARL in the form of Gaussian distribution and assigns certain subtasks to agents with similar roles. RODE [32] further studies how to efficiently discover the roles set by decomposing joint action spaces into restricted role action spaces. HSD 37 is proposed to discover complementary and useful skills for cooperative team play in MARL.
For model generalization in MAS, a generalizable model structure necessary for addressing the problem of varying statc/observationnaatinn space (S/O/A) across tasks. Here we define the agents controlled by MARL policy and agents built-in tasks as entities. As demonstrated in ASN (33, an agent's 's observation o can be constructed as a concatenation of m emiit-obserrations::
ASN 33 && UPDeT [9]
[o..1, [0,,1,0..2, .... m. where 0,,1 is the observation of agent i on itself and environment, and the rest are the observations of agent i on other m 1 entities. Additionally, action space A can be decomposed into two components: A which consists of actions affecting the agent itself or the environment, and Anteractive which contains actions that directly interact with other entities. This alignment between obstryoobsevvations and actions, which is referred to as action semantics, forms the foundation for computing the value or probability of an action based on its aligning entity-bsservation, leading to generalizable model structure across tasks.
Thereafter, by combining ASN with Transformer 29 UPDeT ( develops a type of population- invariant network (PIN) to further improve the model's generalization capability. Specifically, entity-osservations are taken as queries, keys, and values to derive an attention output, as shown in Formula (10) and Formula [II)
Q = Wolo og ..., L. Ki Wx Wxlo oe ..., Lmm V = Wvloo,, oi.,.., om (10)
13

---

of = [pp pm.ml = softmax Q.K.T VV?
(11)
vdk
Subsequently, the attention output fl,1 and pi.g.. = 2,... m are utilized to obtain the Q-value probability of actions in A and Ainteractive respectively. As shown in ODIS [11, UPDeT 9 and D. the pipeline of PIN has been regarded as a foundational framework for the model's generalization across tasks in MARL. Therefore, the PIN is also employed in DT2GS.
B Asymptotic Performance on Single-Task problems
Single-Task
L0tn
1.00
1.00
1.00
0.75
A
MCTO
20%
0.25
0.25
0.30
0.00
22
0.4
0.8
0.0
1.3
0.0
0.2
0.4
0.8
5.0
3.0
0.0
02
041
0.6
3.0
3632
3432
1C3020
3.00
3.00
H499C
190
6:50
0.25
0.25
0:25
1.3
20,90 540g
Ave Averaged
Toa WM Rate
UROUT
0:20
5.5
5.0
10t
Figure 9: Comparison of DT2GS performance with baselines, including UPDeT, ASN, and MAPPO, on Single-Task Problems.
14

---

C Consistent yet Scalable semantics of subtask
DT2GS endows subtasks with consistent yet scalable semantics across tasks. For instance, the semantics of subtask h could involve tempting 2 enemies in the source task while 4 enemies in the target task. The term "consistency" refers to k tempting enemies both the source and target tasks, while "scalability" refers to the changed impact on entities. Figure [T0 Tojillustrates the subtask semantics, which are represented by the subtask's attention on all entities. DT2GS can adapt subtask semantics to different task contexts effectively, reflecting its scalability on subtask semanties across tasks. our experiments, agents paid attention solely to allies as they employed the subtask "disperse" to line up during the t_ period. During the t_2 period, agents initially paid attention on both allies and enemies as they utilized the subtask "hit and run" to focus fire enemies while monitoring their own health and allies' position. Later in t t_2, agents shifted their focus to enemies as they employed the subtask "charge" to concentrate their firepower to end the battle. The last period, t_3, signifies the agents' demise.
observed that
subtask's percentage zero_shor shot
Im
5m 4m
Bn Ser
0.10
1
0.50
0.22
L.2
016
-0.055
1.3
-0.60
enemies
lies
enemies
allics
enemies
10m,o_1im
23m
2hmuoo
are
0.005
3.000
e.es
2
0.000
0.055
t.
1000
-00
enemies
enemies
allies
enemies
Figure 10: Visualization the subtask semantics, which represented by the subtask's 's attention on all entities. We trained model on the source task 8m and evaluated its performance on 6 different marine tasks. To visualize the semantics of subtasks assigned to the agent during an episode, we plotted the subtask attention on all entities over time. The resulting visualization is shown above, where each cell represents the attention weight of a subtask-entig pair at specific timestep. The horizontal axis denotes the entities present in the task, ordered from allies to enemies from left to right, while the vertical axis represents the chronological order timesteps the episode from top to bottom.
15

---

D Generalization on MPE
Zero-Shot
-100
478.80
-200
cp. re.wardd
-300
algo
1239
DT2GS
UPDeT
-500
ASN G
3->+
3-2-4
3-2-6
Figure 11: The figure shows a comparison of zero-shot generalization capability between DT2GS, UPDeT, and ASN G on the physical deception task (Spread) in the multi-agent particle world envi- ronments (MPE) [6]. The horizontal axis represents the source task - the target task. For example, "3-+4" indicates that the source task is set by 3 agents and landmarks, while the target task is set by 4 agents and landmarks. The vertical axis represents the average episode reward when deploying the source model the target task without any finetue. The red, green, and purple histograms correspond to DT2GS, UPDeTT, and ASN_ G, respectively. As we can see, DT2GS significantly outperforms UPDeT and ASN_ G in terms of zero-shot generalization capability, achieving an average episode reward surpass of about 17.05 and 44.32, respectively
E Ablation Study
To investigate the effect of the number of subtasks ng on the model's zero-shot generalization capa- bility, we conducted an ablation study, as shown in Figure [2 Our results indicate that appropriately increasing the number of subtasks can improve the model's zero-shot generalization capability, but setting 7ig 6 leads to degradation due to the limited number of generalizable subtasks across tasks. Additionally, mance, particularly in the zero-shot generalization scenario 8m + 25m. To balance efficiency and effectiveness, set ni 4 for all experiments presented in this paper.
found that increasing nę from 4 to 5 does not significantly improve perfor-
Zero-Shot-Ablation.oog
nk
0.8
0. 89
0.7 34
0.595
0.617
0.609
eval_w win rate
0 0.770
0.484
0.3 67
0.320
0.312
0.2
0.096
0.0
8m 8m, vm
Bm 10m vs_ 11m
8m 25m
Figure 12: Ablation study on the impact subtask number ny on the model's zero-shot generalization capability.
16

---

F List of symbol definitions
Table 1: List of symbol definitions
Symbol
Definitions
Oj
the observation of agent
Oa.j
the entity-observation of agent
entity J all entity ooservtisssss of agent at time
(
o
the entity-observation of agent on itself and environment
gallys
the entity-observation of agent on its jth ally the -uttyy-observation of agent on its jth enemy the action of agent i time
onemy
d
k
the subtask assigned to agent at time t the history latent embedding of agent i at time f
h
the embedding Gaussian Distribution of entity-observation of agent i on entity j at time /
N(H, o
the embedding Gaussian Distribution of observation of agent at time f the observation embedding of agent i at time the adaptive semanties of subtask assigned agent i time the semantics of all actions for agent at time /
G Hyperparameters
Table 2: List of Hyperparameters
Hyperparameters
Value
Algorithms
hidden layer dimension of DT2GS'ss Encoder
8
DT2GS
MLP's hidden layer dimension attention's 's hidden layer dimension
64
DT2GS, UPDeT, ASN, ASN G. MAPPO
64
DT2GS, UPDeT, ASN_G G
attention's 's heads
3
DT2GS, UPDeT, ASN_G DT2GS
number of subtasks
4
optimizer
Adam
DT2GS, UPDeT, ASN, ASN G, MAPPO DT2GS, UPDeT, ASN, ASN G, MAPPO
learning rate of actor and critic
0.0005
To ensure consistency and comparability across our experiments, we aimed to share hyperparameters among algorithms wherever possible. We provide list of the shared hyperparameters in Table [ where the "Algorithms" column indicates which algorithms have corresponding hyperparameters in common.
17

---

H
Pseudocode
Algorithm DT2GS based MAPPO
Input:
The parameters 0 for Scalable Subtask Encoder of actor the parameters 0e for Adaptive Action Decoder of actor æ; the parameters 8 for critic V; the number n. for subtasks Output: The Scalable Subtask Encoder TEn and Adaptive Action Decoder #De for actor n, the
critic V
Initialize En: De: 0V OY,
2: Initialize the total timesteps stepmax interaction with environment; the total timesteps T of an episode; the number of episodes batch_ size for each actor/critic update
3: Initialize step 0
4: while step I stepmmx do
5: set data buffer D =
6: for idr = to batch_size do
empty list 8: initialize actor RNN states l1 ".. 11 for each agent
7:
9:
initialize critic RNN states H1, ....... No for each agent
10:
initialize subtasks kP ..... kn for each agent Initialize subtasks with ng Pg- zero vector for timestep = to do for all agents do
11:
12:
13:
kt. h4,t Tn (of k,-
Call for Scalable Subtask Encoder Module
14:
a = ( k De. " h (s; N
Call for Adaptive Action Decoder Module
end for
Execute actions a', observe re gttl 0et1
17:
T+ = [8 of, h hi, We, at setl oet2)
18:
19:
end for
step+
21:
Compute advantage estimate Å via GAE on
Compute reward-to-go R on
23:
Split trajectory into chunks of length I. for I...,.... T//L do
25:
D = DU
DU (r(1 : T), Al : 1 L), Rl::+ L.)
26:
end for
27:
end for
28:
for mini-batch 1,.., K do
29:
b - random mini-batch from D with all agent data
30:
for each data chunk in the mini batch do update RNN hidden states for TEn and V from first hidden state in data chunk end for
31:
32:
33:
end for
34:
Adam update En 0 on actor loss with data
35:
Adam update 8 on critic loss with data
36:
end while
37:
Return TEn; De: V
18

---

Algorithm 2 Scalable Subtask Encoder
Input:
The parameters En for Scalable Subtask Encoder of actor и, agent i's observation of in timestep agent i's subtask k and actor RNN state , in timestep 1
Output: agent i's subtask k and actor RNN state , in timestep
Compute eebeddding ( = 1, ..., m by Formula
2: Compute observation-ent -enttiinnembeddiing et by Formula
3: Compute actor RNN state h by Formula (4)
4: Compute subtask kę with kę Guubel--Softaaxe (k
5: Return k,, h4.
Algorithm 3 Adaptive Action Decoder
Input:
The parameters te for Adaptive Action Decoder of actor n, agent observation of and subtask &ę in timestep f,
Output: agent action a in timestep f
1: Compute subtask embedding 2 with 2 Embedding(Ak) (k()
2: Compute adaptive subtask semantics of by Formula 7
3: Compute action semanties of by Formula (8)
4: Sample action of from the actions' probability distribution Pr. computed by Formula 9
5: Return o²
19


Reviews:
[{"review_id": "gdCyVOPEq9", "reviewer": "Reviewer_McXb", "summary": "The paper introduces the DT2GS framework to improve the generality of agents in Multi-Agent Reinforcement Learning (MARL) by decomposing a task into generalizable subtasks. The authors use a scalable subtask encoder to identify appropriate subtasks based on historical entity-observation pairs, instead of action-observation pairs, which enhances generalizability. With the subtask selected by the scalable subtask encoder, the paper employs an adaptive subtask semantics with a self-attention mechanism to compute adaptive action semantics that are then passed to an MLP to obtain action distribution and Q values. This adaptive action semantics contains information about the current subtask selected as well as observations from all the entities, making it sufficient for computing action and Q values. The self-attention mechanism used does not have a position embedding, which is suited to the requirement of scalability across tasks regardless of the number of entities and permutation invariance. Empirical results demonstrate that DT2GS possesses sound zero-shot generalization capability across tasks, exhibits sufficient transferability, and outperforms existing methods in both multi-task and single-task problems.", "questions": "Figure 4 only reports the performance of each algorithm trained on the source task but tested on the target task. What is the performance of algorithms on the source task? If one algorithm performs poorly even on the source task, then it makes little sense to test it on a different target task.", "limitations": "The authors did not explicitly discuss the limitations of their work in the paper. However, no major limitations or potential negative societal impacts have been identified in this review.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The proposed framework's structure is well suited to the problem setting, including the scalable subtask encoder and the adaptive subtask semantics.\n2. Experimental results are promising, demonstrating improved performance compared to baseline methods, as well as providing insightful analysis on subtask percentage.", "weaknesses": "1. A more detailed introduction of related work could be helpful for readers if space allows, such as the network design or methodology of ASN.\n2. In Figure 1, the right part uses \"Env Cognition\" and \"Cognition Encoder: MLP\". Adding the word \"cognition\" to appropriate locations in the text around Equation 2 would make Figure 1 match the text better.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your detailed review. We are glad to discuss your concerns one by one. Any further discussion will be appreciated.\n\n1. > **Weakness 1**: A more detailed introduction of related work could be helpful for readers if space allows, such as the network design or methodology of ASN.\n\n   We added a section of related work to the appendix of the revised paper. For details, please see **Weakness 1** of our global response.\n\n2. > **Weakness 2**: In Figure 1, the right part uses \"Env Cognition\" and \"Cognition Encoder: MLP\". Adding the word \"cognition\" to appropriate locations in the text around Equation 2 would make Figure 1 match the text better.\n\n   We added \"cognition\" in our revised paper to ensure consistency between text and figure. Specifically, we modified the content of Line 125 to \"and the observation embedding $e_i^t$, which is also referred to as Env Cognition of agent $i$ in Figure 1, is constructed as: \".\n\n3. > **Question**: Figure 4 only reports the performance of each algorithm trained on the source task but tested on the target task. What is the performance of algorithms on the source task? If one algorithm performs poorly even on the source task, then it makes little sense to test it on a different target task.\n\n   The asymptotic performance of DT2GS, UPDeT, and ASN_G are shown in the table as follows (values before '/' are the algorithms' performance of test winning rate on the source task, and values after '/' are the zero-shot performance of the model on target tasks). As we can see, although DT2GS has the same/similar asymptotic performance as UPDeT/ASN_G on all/most source tasks, its zero-shot generalization performance is significantly higher than UPDeT/ASN_G, achieving an average test winning rate surpass of about 22%/34% over all 8 zero-shot generalization scenarios. \n\n   | source / target     | DT2GS     | UPDeT     | ASN_G         |\n   | ------------------- | --------- | --------- | ------------- |\n   | 3s_vs_4z / 3s_vs_5z | 1 / 0.391 | 1 / 0.203 | 0.109 / 0     |\n   | 2s3z / 3s5z         | 1 / 0.336 | 1/ 0.062  | 0.547 / 0.208 |\n   | 3s5z / 3s5z_vs_3s6z | 1 / 0.875 | 1 / 0.648 | 0.969 / 0.547 |\n   | 8m / 8m_vs_9m       | 1 / 0.734 | 1 / 0.391 | 0.938 / 0.141 |\n   | 8m / 10m_vs_11m     | 1 / 0.609 | 1 / 0.539 | 0.938 / 0.188 |\n   | 8m / 25m            | 1 / 0.367 | 1 / 0.172 | 0.938 / 0.039 |\n   | 8m_vs_9m / 25m      | 1 / 0.484 | 1 / 0.234 | 0.469 / 0.164 |\n   | 8m_vs_9m / 5m_vs_6m | 1 / 0.180 | 1 / 0.070 | 0.469 / 0     |\n"}, {"review_id": "JkVL0nHyfc", "reviewer": "Reviewer_5oAQ", "summary": "This work proposes DT2GS (Decompose a Task inTO a series of Generalizable Subtasks) that addresses multi-agent reinforcement learning in the contexts of zero-shot generalization, transfer, and multi-task. DT2GS learns task-independent subtasks that are characterized by the effects of each agent on itself and other agents when solving the subtask. The proposed method outperforms baselines in SMAC environment in zero-shot, transfer, and multi-task scenarios.", "questions": "**Q1.** The paper seems to lack a rigorous mathematical formulation that captures the definition of a subtask provided on Line 142. Could the authors clarify this aspect with a suitable formulation?\n\n**Q2.** The two-stage self-attention mechanisms utilized in the subtask semantics module do not appear to be immediately intuitive. Could the authors further clarify their ideas from Line 146 to Line 163, particularly focusing on how this structure can prompt the subtask to categorize the impact of the agent on the entities?\n\n**Q3.** It's somewhat perplexing why the MLP (Multi-Layer Perceptron) is designated as \"similarity\", considering it doesn't directly compute the similarities of the embeddings. Could the authors clarify this terminology?\n\n**Q4.** It’s unclear how the learned subtask context is actually used by the policy. A pseudocode \nof the entire process including MAPPO update would be helpful.\n\n**Q5.** The depiction of multiple encoders with a single decoder in Figure 1 is rather confusing. Given that it's a decentralized system, it seems unlikely that each agent shares all input embeddings from all encoders. Does the figure represent the encoder and decoder for the n-th agent, where only the n-th embedding is input into the decoder, excluding the other n-1 agents?\n\n**Q6.** Is the robustness of DT2GS dependent on the choice of n_k=4? Would the method's performance vary significantly with a different choice of n_k?", "limitations": "Despite the authors' affirmation in the checklist regarding limitations, neither the main manuscript nor the appendix seems to provide any explicit discussion of the potential limitations of the proposed method.\n\nA possible limitation of the proposed work could be its inability to generalize to target tasks with a larger distribution shift, especially those that necessitate the implementation of entirely novel subtasks. For instance, an agent that learns four subtasks during training may struggle when the target task requires the execution of a fifth, previously unseen subtask. Moreover, the computational and memory demands imposed by the subtask encoder and semantics module could also present significant challenges, serving as additional potential limitations of this work", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "**S1. Intuitive and motivating formulation of subtasks**\n\nThe concept of defining a subtask based on the influence an agent on other entities is quite interesting and intuitive. This approach effectively translates into the context of multi-agent Reinforcement Learning (RL) problems, much like how the definition of a subtask is grounded on the Markov Decision Process (MDP) dynamics in single-agent RL problems. This parallel offers a refreshing perspective on tackling multi-agent RL scenarios. Also, it is well demonstrated in Figures 5 and 6.\n\n**S2. Generalizing MARL**\n\nThe approach of expanding the scope of multi-agent Reinforcement Learning (RL) to incorporate zero-shot and multitask problems, achieved through the introduction of task-independent subtasks, is notably intriguing. Furthermore, the experiments have been designed to effectively demonstrate these problems.", "weaknesses": "**W1. Lack of clarity**\n\nThis paper could significantly benefit from substantial revisions to enhance its overall clarity.\n\n- Concerning the technical writing, the manuscript suffers from the frequent use of undefined notations and interchangeable deployment of different notations, which create confusion. There are discrepancies between the notations used in equations and those in figures. For instance, the subscripts n, N, and n_a are inconsistently applied, leading to ambiguity. The varying entity notations, which also diverge from those used in ASN [1], generate unnecessary confusion, despite possibly facilitating the writing process. The clarity is further compromised when it is uncertain whether terms like o_i or o^t_i refer to the raw observation or the entity (see Line 73, Line 86, and Figure 1).\n- The authors present the detailed definition of the subtasks after these terms have already been utilized in the text. To improve the flow and comprehension, it could be beneficial to relocate the definitions found on Line 143 to a position before Line 96, where they are first employed.\n- The paper also contains lengthy sentences riddled with grammatical errors (e.g., Line 138 and Line 141), detracting from the readability.\n- The description of the adaptive subtask semantics alongside the attention mechanism, specifically from Line 156 to Line 163, is notably vague and could use clearer explanations.\n\n**W2. Lack of comparisons to previous work**\n\n- The absence of a dedicated section for related work hinders the readers' ability to contextualize this study within the broader scope of the relevant literature. It is strongly suggested that the authors include a comprehensive summary of related work. This should highlight the strengths, weaknesses, differences, and improvements offered by the proposed method in comparison to other techniques in the field. Without this context, DT2GS may risk being perceived as merely a combination of ASN (entity) [1] and UPDeT (attention) [2].\n- The authors appear to have overlooked the need to reference or empirically compare their work to other related studies that also focus on subtasks in Multi-Agent Reinforcement Learning (MARL). Notably missing comparisons to RoDE [3] and LDSA [4] could provide valuable benchmarks.\n\n[1] Wang et al., “Action Semantics Network: Considering the Effects of Actions in Multiagent Systems”, ICLR 2020.\n\n[2] Hu et al., “UPDeT: Universal Multi-agent Reinforcement Learning via Policy Decoupling with Transformers,” ICLR 2021.\n\n[3] Wang et al., “RODE: Learning Roles to Decompose Multi-Agent Tasks,” ICLR 2021. \n\n[4] Yang et al., “LDSA: Learning Dynamic Subtask Assignment in Cooperative Multi-Agent Reinforcement Learning,” NeurIPS 2022.\n\n\n**Acknowledgment Following Rebuttal**\n\nThe author's rebuttal offered comprehensive revisions to the notations for enhanced clarity. Additionally, the inclusion of pseudocode aids in understanding the core concepts. I believe that implementing these proposed changes will significantly strengthen the work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks a lot for your advice on further improving this paper. We would like to discuss them one by one. Any further discussion will be appreciated.\n\n1. > **Weakness 1**: Lack of clarity\n\n   We revised our writing. Specifically :\n\n   (1) We standardized the use of subscripts in our revised paper as follows: \n\n   - $n$ -- the number of agents. \n\n   - $m$ -- the number of entities.\n\n   - $n_{ally}$ -- the number of allies\n\n   - $n_{enemy}$ -- the number of enemies\n\n   In general, the following equation holds: $n=n_{ally}+1, m=n+n_{enemy}$. And we also standardized the use of the following terms to make our paper clearer\n\n   - $o_i$ -- the raw observation of agent $i$\n\n   - $o_i^t$ -- the raw observation of agent $i$ at timestep $t$ (that is, the superscript $t$ denotes timestep)\n\n   - $o_{i,j}$ -- the entity-observation of agent $i$ on entity $j$\n\n   - $o_{i,j}^{t}$ -- the entity-observation of agent $i$ on entity $j$ at timestep $t$\n\n   (2) We added the definition of subtask semantics, which refers to the effects of an agent on entities when performing a given subtask, in the first paragraph of Section 3 (around Line 100).\n\n   (3) We revised the grammatical errors and broke down lengthy sentences into shorter ones that are easier to understand.\n\n2. > **Weakness 2**: Lack of comparisons to previous work\n\n   (1) We added a section of related work to the appendix of the revised paper. For details, please see \"**Weakness 1**\" our global response. Besides, It should be emphasized that DT2GS is not a combination of ASN and UPDeT. The innovation of DT2GS includes proposing a scalable subtask encoder and an adaptive subtask semantic module, both of which are not proposed by previous work.\n\n   (2) We added RODE, ROMA, and HSD into our reference, and added RODE, ROMA, HSD, and LDSA as baselines in our experiments. The results can be seen in \"**Enhancement 1**\" of our global response. We see that DT2GS outperforms all baselines in terms of asymptotic performance.\n\n\n3. > **Question 1**: A formula for subtask\n\n   The mathematical formulation of the subtask is: $k_i^t=\\text{Gumbel-Softmax}(h_i^t)$\n\n   where $h_i^t$ coming from Formula (4) is the history representation of agent $i$. We added the mathematical formulation of the subtask around Line 132 of our paper to make it more clear.\n\n4. >**Weakness 1 (4) && Question 2**: Clarify for subtask semantics\n\n   We made the following improvements to make the part of Adaptive Subtask Semantics more clear to read in our revised paper.\n\n   First, we set the one-hot subtask embedding $z_i^t$ as a formula: $z_i^t=\\text{Embedding}(k_i^t)$\n\n   Then, we modified Formula (5) as follows: $\\phi_i^t=\\text{softmax}(\\frac{\\hat{Q}_i^tK_i^t}{\\sqrt{d_K}})V_i^t, \\quad \\hat{Q}_i^t=W_Qz_i^t$\n\n   where $K_i^t = W_{K}[o_{i, 1}^t, o_{i, 2}^t, ..., o_{i, m}^t], \\quad V_i^t = W_{V}[o_{i, 1}^t, o_{i, 2}^t, ..., o_{i, m}^t]$\n\n   Therefore, we model the adaptive subtask semantics as a weighted sum of all entity-observations to differentiate the impact of the agent on entities.\n\n   At the same time, we kept Formula (5) unchanged: $\\varphi_i^t=\\text{softmax}(\\frac{{Q}_i^tK_i^t}{\\sqrt{d_K}})V_i^t, \\quad {Q}_i^t=W_Qo_i^t$\n\n   Therefore, the calculation of adaptive subtask semantics $\\phi_i^t$ and action semantics $\\varphi_i^t$ can be differentiated by its query matrix $\\hat{Q}_i^t$ and $Q_i^t$.\n\n   Besides, we modified the $Q(a_j|o_i^t)$ in Formula (7) to $Q_{value}(a_j|o_i^t)$ to differentiate the action-value function ($Q_{value}$) and the query matrix ($Q_i^t$ or $\\hat{Q}_i^t$) in the Attention mechanisms. \n\n   In addition, we also added  $Q_{value}$ and $Pr$ in Figure 3 to facilitate readers in understanding the formulas by referring to Figure 3.\n\n\n\n5. > **Question 3**: Using MLP for similarity.\n\n   In fact, the uses of MLP are diverse. MLP is also a kind of embedding, which can describe similarity to a certain extent, for example, there is a similar usage in [1]. Our experimental results also prove the effectiveness of MLP as similarity to a certain extent. \n\n   > [1] Zeng, Kuo-Hao, et al. \"Moving Forward by Moving Backward: Embedding Action Impact over Action Semantics\", ICLR 2022. \n\n\n\n6. > **Question 4**: How subtask is used by policy.\n\n   We believe that the combination of Figure 1 and Figure 3 serves as pseudo code. But to provide readers with a clearer understanding, we also included pseudo code in the revised paper's appendix.\n\n\n\n7. > **Question 5**: Multiple encoders with single decoder in Figure 1.\n\n   Actually, our encoder and decoder can be shared or independent among all agents. Sorry for the inconvenience caused. And we modified Figure 1 in our revised paper to avoid causing such confusion to readers.\n\n\n\n8. > **Question 6**: the choice of $n_k=4$\n\n   $n_k$ is the number of task-independent subtasks across a series of tasks. As is shown in the ablation experiment we conducted in the appendix, increasing $n_k$ appropriately can enhance the diversity of subtasks and thus capture all task-independent subtasks as much as possible. However, an excessive value of $n_k$ can also affect the efficiency of our method, resulting in a poorer performance. Therefore, it's natural that a reasonable choice of $n_k$ matters to DT2GS. \n\n\n\n9. > **Limitations:** Larger distribution shift between source and target tasks.\n\n   DT2GS can capture all of the task-independent subtasks, which are generalizable across different tasks. Therefore, the generalization scenario of DT2GS is that the distribution shift between the source and target task is moderate. And we added this limitation in our revised paper. Furthermore, expanding the applicability of generalization scenarios and reducing computational complexity are the focal points of future work."}, {"review_id": "Ea49KTCwGU", "reviewer": "Reviewer_i2DM", "summary": "This paper focuses on transfer learning of multi-agent reinforcement learning (MARL), by establishing generalizable sub-tasks to enable knowledge reuse. Empirical evidence underscores that the proposed algorithm demonstrates robust zero-shot generalization across a variety of tasks. Furthermore, the algorithm showcases ample transferability and surpasses contemporary methods in both multi-task and single-task problem areas.", "questions": "(a) The paper seems to address the same research problem as reference [36]. It would be beneficial to clarify your unique contributions and innovations in comparison.\n\n(b) For zero-shot generalization to be applicable across multiple tasks, is it necessary for the tasks to be related or to follow the same distribution? If so, it's crucial to articulate this clearly.\n\n(c) An overview of studies on multi-task skill-based (also known as hierarchical) MARL should be provided. More broadly, it would be useful to include work on multi-agent option (also known as skill) discovery in the related works section.\n\n(d) The \"Adaptive Subtask Semantics\" section appears to heavily rely on the Action Semantics Network (ASN). Providing some background on ASN could aid in comprehension.\n\n(e) In the multi-task settings, is the sole difference among tasks the number of entities, for example, shifting from 8m to 10m?\n\n(f) Given its close relevance, it would be beneficial to include [36] as one of the baseline comparisons. (New results are not mandatory.)", "limitations": "The limitation part is not provided.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(a) The design of the algorithm is presented convincingly and with remarkable clarity, providing an adequate level of detail.\n\n(b) The paper presents compelling empirical results that display the algorithm's leading-edge performance in both multi-task and single-task scenarios. The case study outlined in Section 4.3 greatly aids in illuminating its superior capabilities.\n\n(c) The exploration of transferable MARL represents a pivotal research area with substantial potential for practical applications.", "weaknesses": "(a) The paper seems to omit the related work section, which could be pivotal in comprehending the originality and contributions of this study.\n\n(b) As it stands, the algorithm's generalization capacity is restricted to target tasks that can be broken down into a series of task-independent subtasks derived from source tasks. If the tasks are more diverse, it would be beneficial to consider task-specific subtasks as well.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your positive review, insightful feedback, and constructive comments that help improve the quality of the paper! We are glad to answer your questions and would appreciate any further response.\n\n1. > **Weakness (a)**: The paper seems to omit the related work section, which could be pivotal in comprehending the originality and contributions of this study.\n   >\n   > **Question (c)**: An overview of studies on multi-task skill-based (also known as hierarchical) MARL should be provided. More broadly, it would be useful to include work on multi-agent option (also known as skill) discovery in the related works section.\n   >\n   > **Question (d)**: The \"Adaptive Subtask Semantics\" section appears to heavily rely on the Action Semantics Network (ASN). Providing some background on ASN could aid in comprehension.\n\n   We added a section of related work to the appendix of the revised paper. For details, please see \"**Weakness 1**\" of our global response.\n\n2. > **Weakness (b)**: As it stands, the algorithm's generalization capacity is restricted to target tasks that can be broken down into a series of task-independent subtasks derived from source tasks. If the tasks are more diverse, it would be beneficial to consider task-specific subtasks as well.\n\n   Enable the MARL model generalizable across more diverse tasks is a big problem. In the future, we will improve DT2GS by considering both task-independent and task-specific subtasks, to adapt to larger distribution shifts between source and target tasks.\n\n\n\n3. > **Question (a)**: The paper seems to address the same research problem as reference ODIS[36]. It would be beneficial to clarify your unique contributions and innovations in comparison.\n   >\n   > **Question (f)**: Given its close relevance, it would be beneficial to include [36] as one of the baseline comparisons. (New results are not mandatory.)\n\n   DT2GS is quite different from ODIS. Specifically: \n\n   - DT2GS belongs to the online paradigm, while ODIS belongs to the offline paradigm. The different paradigms make the comparison between the two algorithms lack sufficient significance. To demonstrate this, we directly compare the experimental results from the ODIS paper [7] with the experimental results of DT2GS.\n\n     | Task     | DT2GS          | UPDeT       | UPDeT-l     | UPDeT-m     | ODIS       |\n     | -------- | -------------- | ----------- | ----------- | ----------- | ---------- |\n     | 3m       | **100 ± 1.3**  | 98.4 ± 4.0  | 71.0 ± 16.6 | 82.8 ± 16.0 | 98.4 ± 2.7 |\n     | 5m_vs_6m | **93.8 ± 7.0** | 50.0 ± 29.5 | 12.1 ± 12.6 | 17.2 ± 28.0 | 53.9 ± 5.1 |\n\n     As is shown in the table, the results of UPDeT-l, UPDeT-m, and ODIS come from the ODIS paper and are obtained from Expert offline data. We can observe that DT2GS outperforms ODIS in terms of asymptotic performance, especially on the hard task 5m_vs_6m. However, this performance difference cannot be used to prove which algorithm is better, as it is more likely caused by the paradigm to which the algorithms belong. Furthermore, it can be observed that the UPDeT in our paper (the column of UPDeT) is much better than the UPDeT in the ODIS paper (the columns of UPDeT-l and UPDeT-m), which further supports our conclusion.\n\n   - Different problem orientations. ODIS is designed to discover generalizable subtasks (skills) from multi-task offline data. Although DT2GS can interact with multiple tasks simultaneously, its original intention was to discover task-independent subtasks from a single source task and then generalize them to multiple target tasks.\n\n   - Different designs of subtask encoder. ODIS endows the subtask encoder with generalization by a data-driven approach, which prevents the subtask encoder from overfitting to some source task through multi-task offline data. Different from ODIS, DT2GS endows the subtask encoder with generalization by a structure-designed approach, which assigns subtask based on the agent's {subtask, entity-observation} history rather than {action, observation} history. But DT2GS does not conflict with multi-task, and even its generalization capability may be further improved due to simultaneous interaction with multi-tasks.\n\n   - DT2GS further studied the adaptive semantics of the same subtask on different tasks, which is important to further improve the model's generalization capability and was not mentioned in ODIS. The generalizable subtasks should maintain consistent yet scalable semantics across tasks. That is, the same subtask may have different manifestations on different tasks, which can be captured by constructing adaptive semantics of the subtask. \n\n\n\n\n4. > **Question (b)**: For zero-shot generalization to be applicable across multiple tasks, is it necessary for the tasks to be related or to follow the same distribution? If so, it's crucial to articulate this clearly.\n\n   The zero-shot generalization scenario of DT2GS requires a moderate distribution shift between the source and target task. And we articulated this requirement in our revised paper. \n\n\n\n5. > **Question (e)**: In the multi-task settings, is the sole difference among tasks the number of entities, for example, shifting from 8m to 10m?\n\n   In the 2 multi-task problems, including the *marine*-series tasks ({3m, 8m, 8m_vs_9m, 10m_vs_11m}) and *stalker_zealot*-series tasks ({2s3z, 3s5z, 3s5z_vs_3s6z}), the number of entities varies across tasks. However, we have demonstrated in Figure 1 that DT2GS contains sufficient transferability across tasks, in which the number and type of entities simultaneously shift among tasks."}, {"review_id": "528RFaKvih", "reviewer": "Reviewer_NoDQ", "summary": "The paper proposes a new neural network architecture for representing agent policies in multi-agent reinforcement learning. The architecture consists of two parts: (i) a subtask encoder that chooses the subtask to perform based on the subtask-observation history and (ii) a subtask semantics module that chooses the action based on the current observation and the output of the subtask encoder. The user specifies the number of subtasks as a hyperparameter and both modules are trained end-to-end. Experimental evaluation shows that this architecture enables SOTA performance in single-task as well as multi-task settings. Furthermore, the approach enables training policies that generalize to new scenarios in zero-shot and finetuning settings. A qualitative analysis shows that the subtasks learned can be meaningful and interpretable.", "questions": "1. What parts of the architecture are novel and how does it differ from prior work?\n2. Is there any prior work that tries to decompose the policy into a subtask encoder and a subtask semantics module, either in the cooperative multi-agent setting or the single-agent setting?\n3. Would it make sense to consider freezing the subtask semantics module (after training on multiple tasks) during finetuning to perform new tasks that can be accomplished using the same subtasks? This might demonstrate that the learned subtasks generalize to completely new tasks which may require performing the subtasks in a different order.\n4. Is the architecture trainable in a hierarchical way---i.e., train the subtask encoder at a higher level (grouping transitions in some way) than the subtask semantics module?", "limitations": "Limitations are not explicitly discussed and it would be good to include them---e.g., reliance on the assumption that the action space can be decomposed in a certain way (self-action and actions affecting other agents).", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The idea of splitting the policy into two modules to decompose the overall task into subtasks is interesting and seems to work pretty well in the StarCraft environment. The ability to discover interpretable subtasks is an important benefit of the approach.\n2. Zero-shot generalization abilities are improved by this architecture. The learned policy generalizes reasonably well to new scenarios obtained by modifying the original task.\n3. The overall architecture is logical and appears easy to implement.", "weaknesses": "1. The work seems somewhat incremental and I am unsure if there is sufficient reason to believe the generality of the approach w.r.t. different MARL applications and scenarios. The results are purely empirical and it would strengthen the paper to consider another environment different from the StarCraft domain studied in this paper.\n2. Some design choices are not fully justified. For example, hiding the action history from the subtask encoder is mentioned as a reason to enable cross-task generalization. However, the observation history might contain information about the action history in some environments and it is unclear if hiding the action history in such a case will make a difference. In fact, I believe it is important to include an ablation in which the action history is also given to the subtask encoder to show that the proposed method is better.\n3. The architecture seems to use standard techniques and it would really improve the paper to explain the difference w.r.t. prior work and what is novel about this particular architecture---e.g., how it differs from the baselines and what parts of the architecture are borrowed from prior work.\n4. The decomposition of the task into subtasks can also be achieved using a hierarchical approach such as the one in [1]. I believe that it is important to compare the empirical performance of the approach presented in this paper with that of a hierarchical approach.\n\n[1] Yang, Jiachen, Igor Borovikov, and Hongyuan Zha. \"Hierarchical Cooperative Multi-Agent Reinforcement Learning with Skill Discovery.\" International Conference on Autonomous Agents and Multi-Agent Systems. 2020.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your advice on further enhancing this paper. We would like to discuss them one by one and would greatly appreciate any further discussion on the matter.\n\n1. > **Weakness 1**: Experiments on another environment are needed\n\n   We conducted zero-shot experiments on the physical deception task (Spread) in the multi-agent particle world environments (MPE). The experimental results can be seen in \"**Enhancement 2**\" of our global response. It can be seen that DT2GS outperforms UPDeT and ASN_G in terms of zero-shot generalization capability, achieving an average episode reward surpass of about 17.05 and 44.32, respectively.\n\n2. > **Weakness 2**: It's unclear whether hiding the action history can enable cross-task generalization or not. An ablation is needed.\n\n   Perhaps the reviewer misunderstood our approach. Actually, rather than just hiding action history, replacing action history with subtask history is one of the reasons for achieving cross-task generalization. Besides, since different tasks have different action spaces, giving the action history to the subtask encoder will also make DT2GS non-generalizable in terms of model structure. And replacing action history with subtask history not only preserves the agent's abstract (high-level) behavior history but also enables DT2GS generalizable across tasks in terms of model structure.\n\n3. > **Weakness 3 && Question 1**: Which parts of DT2GS are borrowed from prior work and which parts are new?\n\n   To adapt the model structurally to varying observation/action dimensions, DT2GS adopted the population-invariant network (PIN) structure developed in UPDeT, where the proposal of PIN was also promoted by the network design of ASN. \n\n   The innovations in our model architecture mainly include two parts: \n\n   (a) Designed a subtask encoder from the perspective of cross-task generalization by a structure-designed approach. \n\n   (b) Constructed an Adaptive Subtask Semantics module, which was not proposed by any previous work, endowing subtasks with consistent yet scalable semantics across tasks to improve model's generalization capability.\n\n   \n\n4. > **Weakness 4**: Lack of empirical comparison with HSD.\n\n   HSD is proposed to discover complementary skills (subtasks) in MARL, so it is closely related to DT2GS. Therefore, we added HSD[1] into our reference and as one of the baselines in our experiments. The specific results can be seen in \"**Enhancement 1**\" of our global response. It can be observed that DT2GS significantly outperforms HSD in terms of asymptotic performance.\n\n5. > **Question 2**: Is there any prior work that tries to decompose the policy into a subtask encoder and a subtask semantics module?\n\n    DT2GS is the first work that endows subtasks with consistent yet scalable semantics across tasks by the Adaptive Subtask Semantics module. There are several works containing a subtask encoder in its model to construct subtasks/skills/roles/options [1,2,3,4,5]. However, apart from ODIS[1], none of these works designed a subtask encoder from the perspective of cross-task generalization. And different from ODIS which endows the subtask encoder with generalization by a data-driven approach, DT2GS endows the subtask encoder with generalization by a structure-designed approach.\n\n   > [1] Zhang et al., \"Discovering generalizable multi-agent coordination skills from multi-task offline data\", ICLR 2022.\n   >\n   > [2] Yang et al., \"LDSA: Learning dynamic subtask assignment in cooperative multi-agent reinforcement learning\", NeurIPS 2022.\n   >\n   > [3] Wang et al., \"RODE: Learning roles to decompose multi-age\", ICLR 2021.\n   >\n   > [4] Wang et al., \"Roma: Multi-agent reinforcement learning with emergent\", ICML 2020.\n   >\n   > [5] Yang et al., \"Hierarchical cooperative multi-agent reinforcement learning with skill discovery\", AAMAS 2020.\n\n\n\n6. > **Question 3**: Conduct transfer under the setup of freezing the subtask encoder.\n\n   Actually, the zero-shot generalization experiment we conducted froze both the subtask encoder and subtask semantics module, which is more stringent than the experimental setup of just freezing the subtask encoder. Therefore, the zero-shot generalization experiments have demonstrated that the learned subtasks generalize well to completely new tasks which may require performing the subtasks in a different order.\n\n7. > **Question 4**: Is DT2GS trainable in a hierarchical way?\n\n   Yes, we can train the subtask encoder at a higher level than the subtask semantics module. But hierarchically training the model is more commonly used in the setup of sparse reward. With intrinsic rewards, such training way is helpful for temporal credit assignments.\n\n8. > **Limitations**: Assumption that the action space can be decomposed in a certain way\n\n   We added this limitation in our revised paper. In the future, we will improve DT2GS in ways like automatically decomposing the action space to alleviate this limitation."}]

Author rebuttal:
1. > **Weakness 1**: Lack of a section for related work.

   Due to space limitations, we only provided a brief introduction to the related work in the Introduction section. Based on the feedback received, we added a detailed section of related work to the appendix of the revised paper. And the related work is organized according to the following structure: Firstly, we classify the methods of transfer learning across tasks in online MRAL into two categories: Network-design-based methods [1, 2, 3] and Task-embedding-based methods [4, 5, 6]. Secondly, we also summarized the methods of transfer learning across tasks belonging to offline MARL [7] and Curriculum Learning [8, 9] in MRAL. Thirdly, we summarized the related work in MARL focusing on concepts like skills/options/roles [10, 11, 12, 13], which are similar to subtasks studied in our method. Finally, we explained in detail the network design of ASN [2], the action semantics mentioned in ASN, and the population-invariant network (PIN) structure developed in UPDeT, which can help readers to understand the pipeline of our method.

   Previous works focusing on multi-agent transfer learning based on network-design or task-embedding, but lack of efficient knowledge reuse leading to the limited transferability. Our work leverages the knowledge reuse by ensuring semantic consistency between diverse tasks and prevents over-fitting of source tasks, which greatly improves transferability and generalization capability. Results show that DT2GS model can achieve zero-shot generalization capability and more robust transferability, achieving an average transfer speedup of 100×.

   > [1] Hu et al., “UPDeT: Universal Multi-agent Reinforcement Learning via Policy Decoupling with Transformers,” ICLR 2021.
   >
   > [2]  Wang et al., “Action Semantics Network: Considering the Effects of Actions in Multiagent Systems”, ICLR 2020.
   >
   > [3] Agarwal et al., "Learning transferable cooperative behavior in multi-agent teams", ArXiv 2019.
   >
   > [4] Liu et al., "Value function transfer for deep multi-agent reinforcement learning based on n-step returns", IJCAI 2019.
   >
   > [5] Qin et al., "Multi-agent policy transfer via task relationship modeling", ArXiv 2022.
   >
   > [6] Schäfer et al., "Learning task embeddings for teamwork adaptation in multi-agent reinforcement learning", ArXiv 2022.
   >
   > [7] Zhang et al., "Discovering generalizable multi-agent coordination skills from multi-task offline data", ICLR 2022.
   >
   > [8] Long et al., "Evolutionary population curriculum for scaling multi-agent reinforcement learning", ICLR 2019.
   >
   > [9] Wang et al., "From few to more: Large-scale dynamic multiagent curriculum learning", AAAI 2020.
   >
   > [10] Yang et al., “LDSA: Learning Dynamic Subtask Assignment in Cooperative Multi-Agent Reinforcement Learning,” NeurIPS 2022.
   >
   > [11] Wang et al., “RODE: Learning Roles to Decompose Multi-Agent Tasks,” ICLR 2021.
   >
   > [12] Wang et al., "Roma: Multi-agent reinforcement learning with emergent", ICML 2020.
   >
   > [13] Yang et al., "Hierarchical cooperative multi-agent reinforcement learning with skill discovery", AAMAS 2020.

2. > **Enhancement 1**: Added experimental comparisons with some related work.

   We added RODE, ROMA, and HSD into our reference, and added RODE, ROMA, HSD, and LDSA as baselines in our experiments. The new results are shown in the table below.  

   | task         | DT2GS       | UPDeT       | MAPPO       | ASN         | LDSA        | RODE        | ROMA        | HSD         |
   | ------------ | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- |
   | 3s5z_vs_3s6z | 1.00 (0.02) | 1.00 (0.17) | 0.00 (0.23) | 0.84 (0.11) | 0.00 (0.00) | 0.00 (0.00) | 0.01 (0.03) | 0.00 (0.00) |
   | 5m_vs_6m     | 0.94 (0.07) | 0.50 (0.30) | 0.84 (0.06) | 0.06 (0.04) | 0.68 (0.11) | 0.48 (0.22) | 0.67 (0.12) | 0.00 (0.00) |
   | 6h_vs_8z     | 1.00 (0.03) | 0.91 (0.08) | 0.47 (0.18) | 0.00 (0.01) | 0.01 (0.02) | 0.03 (0.03) | 0.15 (0.17) | 0.00 (0.00) |
   | corridor     | 0.95 (0.05) | 0.91 (0.07) | 1.00 (0.02) | 0.09 (0.05) | 0.80 (0.13) | 0.00 (0.00) | 0.01 (0.04) | 0.02 (0.05) |

   The values in the table represent the mean and variance of the test winning rate. As we can see, DT2GS outperforms all baselines. Besides, a more intuitive graph regarding this result can be found in the PDF file we submitted.

3. > **Enhancement 2**: Added experiments on other environments.

   We conducted zero-shot experiments on the physical deception task (Spread) in the multi-agent particle world environments (MPE) [1]. The experimental results are as follows. 

   | source/target | DT2GS           | UPDeT           | ASN_G           |
   | ------------- | --------------- | --------------- | --------------- |
   | 3/2           | -84.44 (5.85)   | -83.61 (2.31)   | -97.43 (8.67)   |
   | 3/3           | -170.88 (23.18) | -206.39 (9.53)  | -212.66 (7.11)  |
   | 3/4           | -265.79 (52.21) | -293.92 (14.58) | -309.32 (22.61) |
   | 3/5           | -371.11 (63.95) | -370.28 (31.05) | -459.51 (19.16) |
   | 3/6           | -472.74 (99.25) | -496.00 (72.95) | -507.66 (21.75) |

   For example, "3/4" in the column of "source/target" indicates that the source task is set by 3 agents and 3 landmarks, while the target task is set by 4 agents and 4 landmarks. And the values in the column of the algorithm represent the mean and variance of the episode rewards. As we can see, DT2GS outperforms UPDeT and ASN\_G in terms of zero-shot generalization capability, achieving an average episode reward surpass of about 17.05 and 44.32, respectively. And a more intuitive bar chart regarding this experiment can be found in the PDF file we submitted. 

   > [1] Lowe, Ryan, et al., "Multi-agent actor-critic for mixed cooperative-competitive environments", NeurIPS 2017.

Meta-review: This paper proposes a new architecture for zero-shot generalisation in multi-agent RL settings. The proposed architectures have two modules: 1) a subtask encoder that chooses what subtask to perform and 2) an action decoder that selects an action conditioned on the subtask encoder's output. The result on StarCraft Multi-Agent Challenge shows that the proposed method not only generalises better to unseen tasks than relevant baselines but also achieve state-of-the-art performances on single-task and multi-task settings. 

The reviewers agreed that the proposed architecture is interesting and reasonable, and the empirical results are quite strong. The reviewers' concerns around the clarity and the lack of discussion on the related work were mostly addressed by the rebuttal. In addition, the authors added more results during the rebuttal period, which further strengthened the result. Since all of the reviewers are not strongly against the paper, I recommend accepting this paper.
Decision: Accept (poster)