Title: Decompose a Task into Generalizable Subtasks in Multi-Agent Reinforcement Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: Multi-Agent Reinforcement Learning

Transfer Learning

Zero-Shot Generalization
Abstract: In recent years, Multi-Agent Reinforcement Learning (MARL) techniques have made significant strides in achieving high asymptotic performance in single task. However, there has been limited exploration of model transferability across tasks. Training a model from scratch for each task can be time-consuming and expensive, especially for large-scale Multi-Agent Systems. Therefore, it is crucial to develop methods for generalizing the model across tasks. Considering that there exist task-independent subtasks across MARL tasks, a model that can decompose such subtasks from the source task could generalize to target tasks. However, ensuring true task-independence of subtasks poses a challenge. In this paper, we propose to \textbf{d}ecompose a \textbf{t}ask in\textbf{to} a series of \textbf{g}eneralizable \textbf{s}ubtasks (DT2GS), a novel framework that addresses this challenge by utilizing a scalable subtask encoder and an adaptive subtask semantic module. We show that these components endow subtasks with two properties critical for task-independence: avoiding overfitting to the source task and maintaining consistent yet scalable semantics across tasks. Empirical results demonstrate that DT2GS possesses sound zero-shot generalization capability across tasks, exhibits sufficient transferability, and outperforms existing methods in both multi-task and single-task problems.

Decompose a Task into Generalizable Subtasks in Multi-Agent Reinforcement Learning

Abstract

In recent years, Multi-Agent Reinforcement Learning (MARL) techniques have made significant strides in achieving high asymptotic performance in single task. However, there has been limited exploration of model transferability across tasks. Training a model from scratch for each task can be time-consuming and expensive, especially for large-scale Multi-Agent Systems. Therefore, it is crucial to develop methods for generalizing the model across tasks. Considering that there exist task-independent subtasks across MARL tasks, a model that can decompose such subtasks from the source task could generalize to target tasks. However, ensuring true task-independence of subtasks poses a challenge. In this paper, we propose to d ecompose a t ask in to a series of g eneralizable s ubtasks (DT2GS), a novel framework that addresses this challenge by utilizing a scalable subtask encoder and an adaptive subtask semantic module. We show that these components endow subtasks with two properties critical for task-independence: avoiding overfitting to the source task and maintaining consistent yet scalable semantics across tasks. Empirical results demonstrate that DT2GS possesses sound zero-shot generalization capability across tasks, exhibits sufficient transferability, and outperforms existing methods in both multi-task and single-task problems.

1 Introduction

In the last few years, many works in MARL field prospers, including value-based algorithms [26, 25, 20, 27, 30, 39, 10, 17], and policy-based algorithms [6, 36, 35, 11, 24, 7]. However, these works mainly focused on the model's asymptotic performance in a single task, neglecting its transferability across tasks. Training an optimal model on a single task requires millions of interactions with the environment [21, 16], particularly when dealing with large-scale Multi-Agent Systems, whereas transferring the model across tasks can reduce training cost by dozens of times [2, 5]. As the number of tasks increases, the reduction in training costs due to model transfer will become even more significant.

Knowledge reuse is a common approach for model generalization across tasks in the MARL field [4]. Recent knowledge reuse methods for online MARL can be roughly classified into two categories: network-design-based methods and task-embedding-based methods. Network-design-based methods [33, 9, 1] implicitly reuse knowledge extracted from the source task to the target task by constructing

universal model structure across tasks by utilizing Population Invariant Structure such as Transformer [9] or GNN [1]. However, it's unclear whether the knowledge extracted from the source task is suitable for the target task. Task-embedding-based methods [19, 23, 14] reuse knowledge by calculating task similarity using learned task embeddings that capture task dynamics. However, accurately mapping task dynamics to task embeddings requires numerous tasks as samples. In a word, current knowledge reuse methods for online MARL still have limitations, that is, inefficient knowledge reuse and reliance on a large number of tasks samples. Additionally, we provide a section for related work in Appendix A to introduce these methods more specifically.

Compared with the common knowledge reused in these methods above, there exists an alternative knowledge named task-independent subtasks that can relieve these two limitations. These taskindependent subtasks, such as "hit and run", "focuse fire", "disperse and line up", etc, are certainly applicable across tasks [41], improving the efficiency of knowledge reuse. Besides, these taskindependent subtasks can be decomposed from few tasks, so as to remove the reliance on a large number of tasks samples. However, ensuring the task-independence of decomposed subtasks is a challenge. Task-independence of subtasks refers to their effectiveness across tasks, which requires two essential properties: (1) avoiding overfitting to the source task , (2) maintaining consistent yet scalable semantics across tasks . In this paper, we propose DT2GS, a novel framework devoted to generalize model across tasks by decomposing subtasks from the source tasks and endowing them with these two properties necessary for task-independence.

The proposed framework, DT2GS, aims to d ecompose a t ask in to a series of g eneralizable s ubtasks, as shown in Figure 1. DT2GS is primarily composed of two parts: the scalable subtask encoder and the adaptive action decoder. The scalable subtask encoder assigns a subtask to each agent based on its {subtask, entity-observation} history rather than the {action, observation} history. This approach helps prevent the process of assigning subtasks from overfitting to the source task. The adaptive action decoder leverages the assigned subtasks and current entity-observations to calculate the specific actions for interacting with environment. Within this decoder, the adaptive subtask semantic module ensures that the assigned subtasks have consistent yet scalable semantics across tasks based on their effects on entities. Based on the experimental results, our framework exhibits several desirable properties as follows:

zero-shot generalization capability : the trained model can be effectively depolyed to multiple target tasks without any fine-tuning;

robust transferability : significantly accelerated model convergence on complex target tasks and achieving an average speedup of 100 × ;

better asymptotic performance : achieved state-of-the-art (SOTA) results on multi-task and single-task problems;

better subtask interpretability : decomposed task-independent subtasks with practical behavioral semantics that is consistent yet scalable across tasks.

2 Preliminary

2.1 Background

In this paper, a fully cooperative multi-agent system (MAS) is described as a decentralized partially observable Markov decision processes (Dec-POMDP) [18], which is defined by a tuple G = < S , A , O, γ, n, A , Ω , R, P > . S is the global state space. A is the action space shared for all agents. O is the shared individual observation space for all agents. γ is the discount factor. n is the number of agents. A = { 1 , ..., n } is the set of agents our algorithm controlled. At each timestep t , agent i obtains an individual observation o t i ∈ O from dynamic environment according to the observation function Ω( s t , i ) . And if we suppose agent i are controlled by policy π , which takes o t i or history individual observations τ t i as input and parameterized by θ i , agent i will select an action a t i according to π ( a t i | o t i ) . Therefore, a joint action A t = ( a 1 , a 2 , ..., a n ) ∈ A will be formed where a i corresponds to agent i . After passing the joint action A t into the environment, a global reward signal r t = R ( s t , A t ) shared by all agents will be received and the environment will transmit into next state s t +1 according to the transition function P ( s t +1 | s t , A t ) : S × A × S → [0 , 1] . The goal of agents is defined as finding the

policy π to maximize the objective function

where γ t ′ R ( s t ′ , A t ′ ) is the discounted return of all agents.

2.2 Generalizable model structure in MARL

For model generalization in MAS, a generalizable model structure is necessary for addressing the problem of varying state/observation/action space ( S /O/ A ) across tasks. Here we define the agents controlled by MARL policy and agents built-in tasks as entities. In this paper, we use n, m, n ally , n enemy denotes the number of agents, entities, allies and enemies, respectively. And the following equation holds: n = n ally + 1 , m = n + n enemy . As demonstrated in ASN [33], an agent's observation o i can be constructed as a concatenation of m entity-observations: o i = [ o i, 1 , o i, 2 , ..., o i,m ] , where o i, 1 is the observation of agent i on itself and environment, and the rest are the observations of agent i on other m -1 entities. Additionally, action space A can be decomposed into two components: A self , which consists of actions affecting the agent itself or the environment, and A interactive , which contains actions that directly interact with other entities. This alignment between entity-observations and actions, which is referred as action semantics, forms the foundation for computing the value or probability of an action based on its aligning entity-observation, leading in a generalizable model structure across tasks.

𝑜

1

𝑡

,

𝑘

1

𝑡-

1

Scalable

Encoder

𝑜𝑛

𝑡

,

𝑘𝑛

𝑡-

1

Scalable

Encoder

…

𝑜𝑛

𝑡

,

𝑘𝑛

𝑡-

1

…

…

[

]

Cognition Encoder: MLP

𝑜 𝑠𝑒𝑙𝑓

𝑜 𝑎𝑙𝑙𝑦

𝑜 𝑒𝑛𝑒𝑚𝑦

𝑘

Product

Env Cognition

GRU

ℎ𝑛

𝑡-

1

ℎ𝑛

𝑡

FC && Gumbel Softmax

𝑘𝑛

𝑡

Entity-obs

Embedding

…

…

Embedding

Adaptive Subtask

Semantics

Action Decision Module

Action/Value

…

…

Adaptive Subtask

Semantics

…

Mean-

Pooling

…

…

…

…

Split

𝑎 𝑎𝑙𝑙𝑦

𝑎 𝑒𝑛𝑒𝑚𝑦

𝑎 𝑠𝑒𝑙𝑓

Adaptive Action Decoder

…

…

[

]

Similarity

𝑒𝑚𝑏

(

𝑜 𝑠𝑒𝑙𝑓

)

𝑒𝑚𝑏

(

𝑜 𝑎𝑙𝑙𝑦

)

𝑒𝑚𝑏

(

𝑜 𝑒𝑛𝑒𝑚𝑦

)

𝑒𝑚𝑏

(

𝑘

)

3 DT2GS Framework

DT2GS primarily consists of two components: the scalable subtask encoder and the adaptive action decoder, as shown in Figure 1. DT2GS begins with the scalable subtask encoder, which assigns a subtask to each agent based on its {subtask, entity-observation} history instead of the {action, observation} history, so as to avoid overfitting to the source tasks. Then the adaptive action decoder calculates the specific actions for interacting with environment by leveraging the assigned subtasks

and current entity-observations. Within this decoder, the adaptive subtask semantic module, which plays a core role, ensures that the assigned subtasks have consistent yet scalable semantics across tasks based on their effects on entities. With the proposed scalable subtask encoder and the adaptive subtask semantic module, DT2GS endows the decomposed subtasks with task-independence, leading model generalizable across tasks. A pseudocode is provided in Appendix H to illustrate the DT2GS framework.

𝑜 𝑖

𝑡-

1

𝑎 𝑖

𝑡-

1

𝑘 𝑖

𝑡-

1

𝑜 𝑖

𝑡

𝑎 𝑖

𝑡

𝑘 𝑖

𝑡

𝑜 𝑖

𝑡

+1

𝑎 𝑖

𝑡

+1

𝑘 𝑖

𝑡

+1

𝑘 𝑖

𝑡-

1

𝑘 𝑖

𝑡

𝑘 𝑖

𝑡

+1

ℎ 𝑖

𝑡-

1

ℎ 𝑖

𝑡

ℎ 𝑖

𝑡

+1

𝑡𝑎𝑠𝑘-𝑑𝑒𝑝𝑒𝑛𝑑𝑒𝑛𝑡

𝑡𝑎𝑠𝑘-𝑖𝑛𝑑𝑒𝑝𝑒𝑛𝑑𝑒𝑛𝑡

{

𝑜 𝑖

,

𝑗

𝑡

+1 }

{

𝑜 𝑖

,

𝑗

𝑡

}

3.1 Scalable Subtask Encoder

An agent's history, which comprises its observations and behavioral history, contains valuable information about its characteristics, such as velocity, attack distance, etc. Therefore, analyzing an agent's history can help us identify the most suitable subtask for the agent to perform. Typically [38], a recurrent neural network, such as LSTM [8] or GRU [3], is employed to construct the agent's history representation from its {action, observation} history. The history representation is then passed to a softmax operator, which samples a one-hot subtask for the agent. However, since actions and observations are task-dependent, this approach may lead to subtask encoder that overfits to the source task, making the source model non-generalizable. Considering that the subtasks history of an agent not only reflects its behavior history, but also is task-independent, and the entity-observations history is more general across tasks than observations history, our scalable subtask encoder assigns a subtask to each agent based on its {subtask, entity-observation} history rather than {action, observation} history, as illustrated in the red solid-line box in Figure 2.

The detail of the scalable subtask encoder is demonstrated in Figure 1 right. By replacing observations with entity-observations, we first design a structure that can obtain fixed-dimensional observation embeddings, regardless of the number of entity-observations contained in the observation. Additionally, since permuting the order of entity-observations in the observation does not change the information[13], we use a Permutation Invariant operator, namely, Gaussian Product, to obtain the observation embedding. Specifically, we first use a MLP parameterized by θ e to embed the entity-observation as an entity-embedding:

and the observation embedding e t i , which is also referred to as Env Cognition of agent i in Figure 1, is constructed as:

Then we utilize a trajectory encoder based on GRU, which is parameterized by θ h , to obtain an agent's history representation. The GRU takes an agent's observation embedding e t i , last subtask k t -1 i and previous hidden state h t -1 i as inputs, and generates a new hidden state h t i as its history representation:

Afterwards, we use the Gumbel-Softmax trick with a categorical reparameterization [12] to assign a subtask k t i to agent i based on its current history representation h t i :

where k t i ∈ R n k is a n k -dimensional one-hot vector and n k is a hyperparameter denoting the total number of subtasks. The Gumbel-Softmax operator allows our process of subtask assignment trainable by the way of gradient backpropagation.

oself

oally1

allynally

oenemy1

oenemyne

Zk

0

↑

Wy

WK

Wo

Matmul&Scale&Softmax

aself

aally

Matmul

qenemy

f

oself

oally

pself

pally1

P

allynally

Φk

oenemy

k

Qvalue /Pr

3.2 Adaptive Subtask Semantics

The generalizable subtasks should maintain consistent yet scalable semantics across tasks. The traditional method of constructing subtask semantics is embedding one-hot subtasks into vector representations based on MLP. However, this approach endows target subtasks with completely identical semantics as source subtasks when deploying the source model to the target tasks, neglecting the difference between source and target tasks so as to restrict the zero-shot generalization of subtask semantics across tasks. To overcome this limitation, our adaptive subtask semantic module leverages the inductive bias that subtask semantics refers to the effects of an agent on entities when it performs a given subtask. This approach endows subtasks with semantics that can adaptively adjust based on the agent's effects on entities, thereby enabling greater zero-shot generalization of subtask semantics across tasks.

Since a subtask's semantics actually refers to the effects of an agent on entities when it performs this subtask, we construct subtask semantics as the weighted sum of entities, where the weight represents the degree of effects. And considering that we have obtained one-hot subtask and entity-observations, we utilize the Attention mechanism [29] without position embedding, which is scalable across tasks regardless of the number of entities and permutation invariant about the order of entity-observaitons in the observation, to obtain the subtask semantics, as illustrated in Figure 3. Specifically, we take one-hot subtask embedding

as the query and entity-observations' embedding as the keys as well as values: ˆ Q t i = W Q z t i , K t i = W K o t i , V t i = W V o t i , where W Q , W K , W V are learnable parameters and o t i = [ o t i, 1 , o t i, 2 , ..., o t i,m ] . Then we construct adaptive subtask semantics ϕ t i as follows:

where d K is the feature dimension of K t i . Additionally, to make the process of obtaining actions scalable across tasks, we extended the model structure of ASN [33]. Since the alignment between actions and entity-observations may discrepant from the prior, we construct the adaptive action semantics φ t i = [ φ t i, 1 , φ t i, 2 , ..., φ t i,m ] by utilizing the Self-Attention mechanism taking the entityobservations as inputs:

Subsequently, we calculate the value or probability of each action in current observation by comparing the similarity of subtask semantics with action semantics:

where similarity is a trainable MLP taking the concatenate of ϕ t i and φ t i,j as inputs.

4 Experiments

4.1 Experimental Setup

We evaluated the performance of DT2GS on the StarCraft Multi-Agent Challenge (SMAC) [22] and the multi-agent particle world environments (MPE) [16] (shown in Appendix D). SMAC contains several tasks that are similar but different, such as the marine -series tasks ({3m, 8m, 8m_vs_9m, 10m_vs_11m, 25m, 27m_vs_30m}) and the stalker_zealot -series tasks (2s3z, 3s5z, 3s5z_vs_3s6z). Besides, changing the number of agents and landmarks in MPE can also form a series of tasks. These tasks met our requirements of cross-task generalization, allowing us to evaluate the ability of DT2GS to generalize across different tasks.

In the generalization capability part of experiments, we selected ASN [33] and UPDeT [9] as baselines. ASN was chosen because it promotes the development of universal models across tasks in MARL. For the sake of fairness in comparison, we make ASN generalizable across tasks by utilizing the attention mechanism and use "ASN_G" to denote this generalizable ASN. UPDeT was selected because it constructs a universal model across tasks via policy decoupling with self-attention, which is also used in the adaptive subtask semantic module of DT2GS in Sec 3.2. In addition, we implemented DT2GS, UPDeT, and ASN_G based on MAPPO [40], which is considered SOTA in on-policy MARL.

In the asymptotic performance part of experiments, we added another five baselines for comparison: MAPPO[40], LDSA [38], ROMA [31], RODE [32], and HSD [37]. LDSA, ROMA, RODE, and HSD focus on concepts like skills/options/roles in MARL, which are similar to subtasks studied in our method.

Additionally, We set the number of subtasks n k to 4 and averaged all results over 4 random seeds. And the experiments are arranged as follows: Firstly, we evaluated the zero-shot generalization capability of DT2GS across tasks. Secondly, we analyzed the practical semantics of subtasks on the marine -series tasks. Thirdly, we conducted several experiments to exhibit the transferability of DT2GS across tasks. Finally, we demonstrated that DT2GS achieves SOTA performance on multi-task and most single-task problems in terms of asymptotic performance.

4.2 Zero-Shot Generalization across Tasks

In this section, we designed 8 different zero-shot generalization scenarios, where each scenario includes a target task that is more difficult than the source task. That is to say, the relationship from the source task to the target task is extrapolated. For example, the target task may be larger in scale (like 8m → 25m) or contain a greater disparity in military strength than the source task (3s_vs_4z → 3s_vs_5z). We deployed the model trained on the source tasks to the target tasks without any finetune. As shown in Figure 4, DT2GS significantly outperforms UPDeT and ASN_G in terms of zero-shot generalization capability, achieving an average test winning rate surpass of about 22% and 34%, respectively, over all 8 zero-shot generalization scenarios.

Zero-Shot

0.875

algo

T

0.8

DT2GS

0.734

UPDeT

0.648

eval_win_rate

0.6

609.0

ASN_G

0.547

0.539

0.484

0.4

0.391

0.391

0.367

0.336

0.234

0.2

0.203

0.208

0.141

0.188

0.172

0.164

0.180

T

0.062

00

0.0

0.000

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

4.3 Analysis of Subtask Semantics

In this section, we analyzed the practical semantics of subtasks based on the marine -series tasks. We first trained a DT2GS model from scratch on the 8m task and then deployed it to 8m and 10m_vs_11m, as shown in Figure 5 left. In the 8m and 10m_vs_11m tasks, the DT2GS agents initially utilized subtask_1 to scatter the formation and advance towards the enemy at t = 1 . When the enemies reached the attack range of DT2GS agents at t = 6 , most of these agents selected subtask_2 to focus fire on enemies while also paying attention to their own health in order to move back when in danger, thereby avoiding the decrease in firepower caused by personnel reduction. And other agents chose subtask_4 to slightly adjust their relative position with the enemies to seek a better position for firepower output. At t = 13 , some agents sensed that their allies' health was too low while their own health was safe enough. These agents switched subtask from subtask_2 to subtask_3, which means charging. And the specific behavior was to attack while moving closer to the enemies' position to attract hatred so as to avoid the decrease in firepower caused by the death of allies. At the end of the battle (after t = 21 ), all agents changed their subtasks to subtask_3 to concentrate their firepower to end the battle. We observed that under the subtask of charge, even if an agent was close to death, it would not move back to disperse hatred as overall withdrawal would lead to failed combat.

Furthermore, we conducted zero-shot generalization experiments on additional marine tasks, and the corresponding change process of subtask percentage is shown in Figure 6. We observed that the change process of each subtask's percentage in the source task (8m) is quite similar to that in the target tasks (10m_vs_11m, 5m_vs_6m, 8m_vs_9m, 25m, and 27m_vs_30m). This result demonstrates that the subtasks decomposed by DT2GS from the source task (8m) are task-independent across tasks and

disperse

hit and run

charge

pos adjust

8m

10m_vs_11m

t=1

t=7

t=13

t=21

t=24

dead

dead

8m (win)

1.0

subtask_percent

0.8

disperse

hit and run

charge

0.2

pos adjust

0.0

0

5

10

15

20

25

timestep

10m_vs_11m (win)

1.0

subtask_percent

0.8

disperse

hit and run

0.4

charge

pos adjust

0.2

0.0

0

5

10

20

25

timestep

therefore effective for the target tasks. Both results from Figure 5 and Figure 6 further suggest that DT2GS policy has sound zero-shot generalization capability by decomposing the task into a series of task-independent subtasks.

subtask'spercentagewhen8m'szero_shot

8m (win)

5m_vs_6m (win)

8m_vs_9m (win)

1.0

1.0

1.0

subtask_percent

0.8

subtask_percent

0.8

subtask_percent

0.8

disperse

disperse

disperse

0.6

hit and run

0.6

hit and run

0.6

hit and run

0.4

charge

0.4

charge

0.4

charge

pos adjust

pos adjust

pos adjust

0.2

0.2

0.2

0.0

0.0

0.0

0

5

10

15

20

25

0

5

10

15

20

25

30

0

5

10

15

20

25

timestep

timestep

timestep

10m_vs_11m (win)

25m (win)

27m_vs_30m(lose)

1.0

1.0

1.0

subtask_percent

0.8

subtask_percent

0.8

subtask_percent

0.8

disperse

disperse

disperse

0.6

hit and run

0.6

hit and run

0.6

hit and run

0.4

charge

0.4

charge

0.4

charge

pos adjust

pos adjust

pos adjust

0.2

0.2

0.2

0.0

0.0

0.0

0

5

10

15

20

25

0

5

10

15

20

25

0

5

10

15

20

25

30

timestep

timestep

timestep

4.4 Transferability across Tasks

Figure 7 (a) illustrates four scenarios that we designed to evaluate the transferability of DT2GS policy. In all four scenarios, the relationships from the source task to the target task are extrapolation, including an increase in entity types (3s5z_vs_3s6z → 1c3s5z), an increase in the number of enemies (3s5z → 3s5z_vs_3s6z and 3s_vs_4z → 3s_vs_5z), as well as an increase in the total scale of entities (8m_vs_9m → 27m_vs_30m).

As we can see, DT2GS_finetune demonstrated significantly efficient convergence as well as higher and more stable asymptotic performance compared with UPDeT_finetune and ASN_G_finetune in all transfer scenarios. Notably, in the scenario where the source task was 3s5z (easy) and the target task was 3s5z_vs_3s6z (superhard), DT2GS_finetune achieved optimal performance with only 6400 steps interaction with environment, whereas learning from scratch required at least 6 million steps. Furthermore, compared to other baselines of learning from scratch, DT2GS_finetune accelerated

the model's convergence on target tasks by an average of 100 × . In conclusion, DT2GS_finetune exhibited better performance than baselines on the following evaluation metrics [28, 4]: jumpstart, time to threshold and asymptotic performance, demonstrating significantly sufficient transferability.

FineTune

Single-Task

1.00

3s52_vs_3s6z~>1c3s5z

1.00

3s5z>355z_vs_3s6z

1.00

3s5z_vs_3s6z

1.00

5m_vs_6m

N0.50

.5

0.00

UPDeT_finetune

8m_vs_9m>27m_vs_30m

Le5

0.00

0°t

3s_vs_4z>35s_vs_5z

1.5

steps

2.0

2.5

1e5

1.00

0.00

0.0

0.2

0.4

6h_vs_8z

steps

0.6

0.8

LOT

0.00

1.00

0.0

DT2GS

ASN

0.2

0.4

corridor

steps

0.6

0.8

1e7

0.00

0.0

UPDeT

MAPPO

ASN

0.2

0.4

steps

0.0

0.5

1.0

1.5

steps

2.0

2.5

0'E

1e5

0.00

0.0

0.2

0.4

steps

0.6

8°0

L0

0.00

0.0

RODE

ROMA

LDSA

0.2

0.4

steps

0.6

8'0

0

(a) Transferability

(b) Single-Task

4.5 Performance on Multi-task and Single-task

In this section, we first designed 2 multi-task problems, including the marine -series tasks ({3m, 8m, 8m_vs_9m, 10m_vs_11m}) and stalker_zealot -series tasks ({2s3z, 3s5z, 3s5z_vs_3s6z}), to demonstrate the representational capacity of DT2GS. In each multi-task problem, the policy interacted synchronously with multiple tasks that make up this multi-task to collect data, which was shuffled and used to update the policy. As shown in Figure 8, DT2GS exhibited better learning efficiency and more stable asymptotic performance compared with UPDeT and ASN_G.

3m

8m

8m_vs_9m

1.00

1.00

1.00

DT2GS

0.50

r

DT2GS

0.50

0.25

DT2GS

0.50

ASN_G

0.00

ASN_G

0.00

ASN_G

0.00

0.0

0.2

10m_vs_11m

0.4

steps

0.6

0.8

1.0

1e7

0.0

0.2

0.4

ste3s

0.6

0.8

1.0

1e7

0.0

0.2

0.4

stsps

0.6

0.8

1.0

1e7

Test Win_Rate

1.00

Test_Win_Rate

1.00

1.00

DT2GS

0.75

0.00

UPDeT

ASN_G

0.50

0.25

0.00

UPDeT

DT2GS

ASN_G

0.50

0.25

0.00

DT2GS

ASN_G

0°0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

3s5z_vs_3s6z

steps

1e7

['3m', *8m', *8m_vs_9m', *10m_vs_11m’]

steps

1e7

[2s3z, '3s5z", '3s5z_vs_3s6z']

steps

1e7

1.00

DT2GS

1.00

1.00

UPDeT

ASNG

0.50

DT2GS

0.50

DT2GS

0.25

0.25

UPDeT

UPDeT

0.00

0.00

ASN_G

0.00

ASN_G

0.0

0.2

0.4

0.6

0.8

1.0

le7

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

1e7

steps steps

1e7

steps

Subsequently, we also compared DT2GS with baselines on single-task scenarios. We selected four representative scenarios, including superhard / hard tasks, as shown in Figure 7 (b), while performance on other scenarios is presented in the Appendix B. Our evaluation metrics included learning efficiency and final asymptotic performance. Compared to MAPPO, DT2GS significantly improved both metrics, particularly in superhard tasks such as 3s5z_vs_3s6z and 6h_vs_8z. In addition, compared to other baselines based on MAPPO, including UPDeT and ASN, DT2GS consistently outperformed them in terms of learning efficiency and asymptotic performance, on average.

5 Conclusion

Model generalization has emerged as a promising approach to reduce training costs. In this paper, we proposed DT2GS, an effective approach for addressing the problem of model generalization across tasks in the MARL field. Our insight is that task-independent subtasks exist across tasks, making it possible to generalize the model across tasks. Based on this insight, we assumed that the model can be generalized to target tasks if it can decompose task-independent subtasks from source tasks. The challenge then becomes ensuring that the subtasks we decompose from the source task are truly task-independent. Regarding this issue, we proposed two properties that enable task-independence of subtasks: (1) avoiding overfitting to the source task, (2) maintaining consistent yet scalable semantics across tasks. Then we proposed DT2GS to endow the subtasks with these two properties by introducing the scalable subtask encoder and the adaptive subtask semantic module, respectively. Empirical results demonstrated that DT2GS can decompose tasks into a series of generalizable subtasks, leading to a generalizable MARL policy. Nevertheless, it would be beneficial to consider task-specific subtasks as well when there is a significant distribution shift between source and target tasks. In our future work, we will focus on expanding the generalization scenarios of DT2GS to address this limitation.

References

Akshat Agarwal, Sumit Kumar, and Katia Sycara. Learning transferable cooperative behavior in multi-agent teams. arXiv preprint arXiv:1906.01202 , 2019.

Georgios Boutsioukis, Ioannis Partalas, and Ioannis Vlahavas. Transfer learning in multi-agent reinforcement learning domains. In Recent Advances in Reinforcement Learning: 9th European Workshop, EWRL 2011, Athens, Greece, September 9-11, 2011, Revised Selected Papers 9 , pages 249-260. Springer, 2012.

Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoderdecoder for statistical machine translation. arXiv preprint arXiv:1406.1078 , 2014.

Felipe Leno Da Silva and Anna Helena Reali Costa. A survey on transfer learning for multiagent reinforcement learning systems. Journal of Artificial Intelligence Research , 64:645-703, 2019.

Sabre Didi and Geoff Nitschke. Multi-agent behavior-based policy transfer. In Applications of Evolutionary Computation: 19th European Conference, EvoApplications 2016, Porto, Portugal, March 30-April 1, 2016, Proceedings, Part II 19 , pages 181-197. Springer, 2016.

Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Proceedings of the AAAI conference on artificial intelligence , volume 32, 2018.

Qingxu Fu, Tenghai Qiu, Jianqiang Yi, Zhiqiang Pu, and Shiguang Wu. Concentration network for reinforcement learning of large-scale multi-agent systems. arXiv preprint arXiv:2203.06416 , 2022.

Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation , 9(8):1735-1780, 1997.

Siyi Hu, Fengda Zhu, Xiaojun Chang, and Xiaodan Liang. Updet: Universal multi-agent reinforcement learning via policy decoupling with transformers. arXiv preprint arXiv:2101.08001 , 2021.

Shariq Iqbal, Christian A Schroeder De Witt, Bei Peng, Wendelin Böhmer, Shimon Whiteson, and Fei Sha. Randomized entity-wise factorization for multi-agent reinforcement learning. In International Conference on Machine Learning , pages 4596-4606. PMLR, 2021.

Shariq Iqbal and Fei Sha. Actor-attention-critic for multi-agent reinforcement learning. In International conference on machine learning , pages 2961-2970. PMLR, 2019.

Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.01144 , 2016.

HAO Jianye, Xiaotian Hao, Hangyu Mao, Weixun Wang, Yaodong Yang, Dong Li, Yan Zheng, and Zhen Wang. Boosting multiagent reinforcement learning via permutation invariant and permutation equivariant networks. In The Eleventh International Conference on Learning Representations .

Yong Liu, Yujing Hu, Yang Gao, Yingfeng Chen, and Changjie Fan. Value function transfer for deep multi-agent reinforcement learning based on n-step returns. In IJCAI , pages 457-463. Macao, 2019.

Qian Long, Zihan Zhou, Abhibav Gupta, Fei Fang, Yi Wu, and Xiaolong Wang. Evolutionary population curriculum for scaling multi-agent reinforcement learning. arXiv preprint arXiv:2003.10423 , 2020.

Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, OpenAI Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-competitive environments. Advances in neural information processing systems , 30, 2017.

Shuang Luo, Yinchuan Li, Jiahui Li, Kun Kuang, Furui Liu, Yunfeng Shao, and Chao Wu. S2rl: Do we really need to perceive all states in deep multi-agent reinforcement learning? arXiv preprint arXiv:2206.11054 , 2022.

Frans A Oliehoek and Christopher Amato. A concise introduction to decentralized POMDPs . Springer, 2016.

Rongjun Qin, Feng Chen, Tonghan Wang, Lei Yuan, Xiaoran Wu, Zongzhang Zhang, Chongjie Zhang, and Yang Yu. Multi-agent policy transfer via task relationship modeling. arXiv preprint arXiv:2203.04482 , 2022.

Tabish Rashid, Gregory Farquhar, Bei Peng, and Shimon Whiteson. Weighted qmix: Expanding monotonic value function factorisation for deep multi-agent reinforcement learning. Advances in neural information processing systems , 33:10199-10210, 2020.

Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. In International conference on machine learning , pages 4295-4304. PMLR, 2018.

Mikayel Samvelyan, Tabish Rashid, Christian Schroeder De Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr, Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. arXiv preprint arXiv:1902.04043 , 2019.

Lukas Schäfer, Filippos Christianos, Amos Storkey, and Stefano V Albrecht. Learning task embeddings for teamwork adaptation in multi-agent reinforcement learning. arXiv preprint arXiv:2207.02249 , 2022.

Wenling Shang, Lasse Espeholt, Anton Raichuk, and Tim Salimans. Agent-centric representations for multi-agent reinforcement learning. arXiv preprint arXiv:2104.09402 , 2021.

Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In International conference on machine learning , pages 5887-5896. PMLR, 2019.

Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al. Valuedecomposition networks for cooperative multi-agent learning. arXiv preprint arXiv:1706.05296 , 2017.

Ming Tan. Multi-agent reinforcement learning: Independent vs. cooperative agents. In Proceedings of the tenth international conference on machine learning , pages 330-337, 1993.

Matthew E Taylor and Peter Stone. Transfer learning for reinforcement learning domains: A survey. Journal of Machine Learning Research , 10(7), 2009.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. Qplex: Duplex dueling multi-agent q-learning. arXiv preprint arXiv:2008.01062 , 2020.

Tonghan Wang, Heng Dong, Victor Lesser, and Chongjie Zhang. Roma: Multi-agent reinforcement learning with emergent roles. arXiv preprint arXiv:2003.08039 , 2020.

Tonghan Wang, Tarun Gupta, Anuj Mahajan, Bei Peng, Shimon Whiteson, and Chongjie Zhang. Rode: Learning roles to decompose multi-agent tasks. arXiv preprint arXiv:2010.01523 , 2020.

Weixun Wang, Tianpei Yang, Yong Liu, Jianye Hao, Xiaotian Hao, Yujing Hu, Yingfeng Chen, Changjie Fan, and Yang Gao. Action semantics network: Considering the effects of actions in multiagent systems. arXiv preprint arXiv:1907.11461 , 2019.

Weixun Wang, Tianpei Yang, Yong Liu, Jianye Hao, Xiaotian Hao, Yujing Hu, Yingfeng Chen, Changjie Fan, and Yang Gao. From few to more: Large-scale dynamic multiagent curriculum learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 7293-7300, 2020.

Yihan Wang, Beining Han, Tonghan Wang, Heng Dong, and Chongjie Zhang. Dop: Offpolicy multi-agent decomposed policy gradients. In International Conference on Learning Representations , 2020.

Ying Wen, Yaodong Yang, Rui Luo, Jun Wang, and Wei Pan. Probabilistic recursive reasoning for multi-agent reinforcement learning. arXiv preprint arXiv:1901.09207 , 2019.

Jiachen Yang, Igor Borovikov, and Hongyuan Zha. Hierarchical cooperative multi-agent reinforcement learning with skill discovery. arXiv preprint arXiv:1912.03558 , 2019.

Mingyu Yang, Jian Zhao, Xunhan Hu, Wengang Zhou, Jiangcheng Zhu, and Houqiang Li. Ldsa: Learning dynamic subtask assignment in cooperative multi-agent reinforcement learning. Advances in Neural Information Processing Systems , 35:1698-1710, 2022.

Yaodong Yang, Jianye Hao, Ben Liao, Kun Shao, Guangyong Chen, Wulong Liu, and Hongyao Tang. Qatten: A general framework for cooperative multiagent reinforcement learning. arXiv preprint arXiv:2002.03939 , 2020.

Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo in cooperative multi-agent games. Advances in Neural Information Processing Systems , 35:24611-24624, 2022.

Fuxiang Zhang, Chengxing Jia, Yi-Chen Li, Lei Yuan, Yang Yu, and Zongzhang Zhang. Discovering generalizable multi-agent coordination skills from multi-task offline data. In The Eleventh International Conference on Learning Representations , 2023.

A Related Work

Transfer learning across tasks in online MARL Transfer learning across tasks is a promising approach to accelerate the convergence of models, especially in large-scale MAS. The first method of transfer learning in online MARL domains is BITER [2], which uses the joint policy learned in the source task to bias the initial policy of agents in the target tasks. After more than a decade of development, some methods have been proposed in this field. In this regard, we classify these methods into two categories based on their characteristics, namely Network-design-based methods and Task-embedding-based methods.

The Network-design-based methods implicitly reuse knowledge from source tasks by constructing universal model structure across tasks using Population Invariant Structure such as Transformer [9] or GNN [1]. Since the existence of varying state/observation/action space across tasks, the Network-design-based methods are the foundation of all other methods. And the Task-embeddingbased methods are knowledge transfer by calculating task similarity using learned task embeddings that capture task dynamics [2, 5, 14, 23, 19]. For example, MATTAR [19] models the target task representation as a weighted sum of a series of source task representations and decodes the task representation to the policy weight that is suitable for the task by utilizing a hypernetwork. And MATE [23] models task representation by reconstructing the task dynamics, so as to achieve task differentiation and fast adaptation of policy on tasks.

Previous works focused on multi-agent transfer learning based on network-design or task-embedding, but lacked of efficient knowledge reuse, leading to limited generalization capability. Our work (DT2GS) leverages knowledge reuse by ensuring semantic consistency and scalability between diverse tasks and preventing over-fitting of source tasks, which greatly improves transferability and even zero-shot generalization capability.

Transfer learning across tasks in other paradigms There are some methods dedicated to transfer learning across tasks in MARL belonging to other paradigms. For example, ODIS [41], belonging to the offline RL paradigm, discovers generalizable skills from multi-task offline data to enhance the model's generalization capability to unseen tasks. Besides, EPC [15] and DyMA-CL [34] fulfill knowledge transfer across tasks by way of Curriculum Learning.

Skills/Options/Roles/Subtasks in MARL There are some works in MARL focusing on concepts like skills/options/roles, which are similar to subtasks studied in our method. LDSA [38] learns dynamic subtask assignments and constructs different policies for different subtasks, so enabling agents to possess specific abilities to handle different subtasks. ROMA [31] introduces the role concept into MARL in the form of Gaussian distribution and assigns certain subtasks to agents with similar roles. RODE [32] further studies how to efficiently discover the roles set by decomposing joint action spaces into restricted role action spaces. HSD [37] is proposed to discover complementary and useful skills for cooperative team play in MARL.

ASN [33] && UPDeT [9] For model generalization in MAS, a generalizable model structure is necessary for addressing the problem of varying state/observation/action space ( S /O/ A ) across tasks. Here we define the agents controlled by MARL policy and agents built-in tasks as entities. As demonstrated in ASN [33], an agent's observation o i can be constructed as a concatenation of m entity-observations: o i = [ o i, 1 , o i, 2 , ..., o i,m ] , where o i, 1 is the observation of agent i on itself and environment, and the rest are the observations of agent i on other m -1 entities. Additionally, action space A can be decomposed into two components: A self , which consists of actions affecting the agent itself or the environment, and A interactive , which contains actions that directly interact with other entities. This alignment between entity-observations and actions, which is referred to as action semantics, forms the foundation for computing the value or probability of an action based on its aligning entity-observation, leading to a generalizable model structure across tasks.

Thereafter, by combining ASN with Transformer [29], UPDeT [9] develops a type of populationinvariant network (PIN) to further improve the model's generalization capability. Specifically, entity-observations are taken as queries, keys, and values to derive an attention output, as shown in Formula (10) and Formula (11).

Subsequently, the attention output φ t i, 1 and φ t i,j , j = 2 , ..., m are utilized to obtain the Q-value or probability of actions in A self and A interactive , respectively. As shown in ODIS [41], UPDeT [9] and [1], the pipeline of PIN has been regarded as a foundational framework for the model's generalization across tasks in MARL. Therefore, the PIN is also employed in DT2GS.

B Asymptotic Performance on Single-Task problems

Single-Task

3m

8m_vs_9m

10m_vs_11m

1.00

1.00

1.00

Rate

0.75

Rate

0.75

Rate

0.75

Win

0.50

MAPPO

UPDeT

0.50

MAPPO

UPDeT

Win

0.50

UPDeT

MAPPO

0.25

ASN

0.25

ASN

0.25

MMmm

ASN

0.00

DT2GS

0.00

DT2GS

0.00

DT2GS

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

steps

1e7

steps

1e7

steps

1e7

2s3z

3s5z

1c3s5z

1.00

1.00

1.00

Rate

0.75

0.75

0.75

Win

0.50

UPDeT

MAPPO

Win

0.50

MAPPO

UPDeT

0.50

MAPPO

UPDeT

Test

0.25

一

ASN

ASN

0.25

一

ASN

DT2GS

DT2GS

DT2GS

0.00

0.00

0.00

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

3s_vs_5z

steps

1e7

2c_vs_64zg

steps

1e7

Averaged

steps

1e7

1.00

1.00

1.0

Rate

0.75

0.50

UPDeT

MAPPO

Rate

Win

0.75

0.50

UPDeT

MAPPO

0.8

0.6

0.4

0.25

ASN

DT2GS

ASN

DT2GS

0.2

0.00

0.00

0.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

steps

1e7

steps

1e7

C Consistent yet Scalable semantics of subtask

DT2GS endows subtasks with consistent yet scalable semantics across tasks. For instance, the semantics of subtask k could involve tempting 2 enemies in the source task while 4 enemies in the target task. The term "consistency" refers to k tempting enemies in both the source and target tasks, while "scalability" refers to the changed impact on entities . Figure 10 illustrates the subtask semantics, which are represented by the subtask's attention on all entities. We observed that DT2GS can adapt subtask semantics to different task contexts effectively, reflecting its scalability on subtask semantics across tasks. In our experiments, agents paid attention solely to allies as they employed the subtask "disperse" to line up during the t_1 period. During the t_2 period, agents initially paid attention on both allies and enemies as they utilized the subtask "hit and run" to focus fire on enemies while monitoring their own health and allies' position. Later in t_2, agents shifted their focus to enemies as they employed the subtask "charge" to concentrate their firepower to end the battle. The last period, t_3, signifies the agents' demise.

t_2

t_2

t_2

t_2

t_2

t_2

allies

enemies

allies

enemies

allies

enemies

allies

enemies

allies

enemies

allies

enemies

subtask'spercentage when8m'szero_shot

8m

5m_vs_6m

8m_vs_9m

0.10

0.150

0.10

0.08

H

0.125

0.08

0.100

0.06

0.06

0.075

0.04

- 0.04

-0.050

10m_vs_11m

25m

27m_vs_30m

0.08

0.035

0.030

0.030

0.06

0.025

0.025

0.020

0.020

0.04

0.015

0.015

- 0.02

- 0.010

0.010

D Generalization on MPE

Zero-Shot

0

-100

-84 44-83. 62

-9743

170.89

-200

20g.39.212.66

ep_rewards

26

291.93 305.32

-300

algo

-400

DT2GS

45.5

UPDeT

472.74

1496.0050f.67

-500

ASN_G

3-->2

3-->3

3-->4

3-->5

3-->6

E Ablation Study

To investigate the effect of the number of subtasks n k on the model's zero-shot generalization capability, we conducted an ablation study, as shown in Figure 12. Our results indicate that appropriately increasing the number of subtasks can improve the model's zero-shot generalization capability, but setting n k = 6 leads to degradation due to the limited number of generalizable subtasks across tasks. Additionally, we found that increasing n k from 4 to 5 does not significantly improve performance, particularly in the zero-shot generalization scenario 8m → 25m. To balance efficiency and effectiveness, we set n k = 4 for all experiments presented in this paper.

Zero-Shot-Ablation_on_nk

T

0.828

nk

0.8

T

0.7340.142

T

0.789

2

0.695

3

eval_win_rate

0.6

0.617

0.570

0.609

4

５

0.484

6

0.4

0.359

0.367

0.320

0.312

0.2

0.180

0.086

0.0

8m→8m_vs_9m

8m→10m_vs_11m

8m→25m

F List of symbol definitions

G Hyperparameters

To ensure consistency and comparability across our experiments, we aimed to share hyperparameters among algorithms wherever possible. We provide a list of the shared hyperparameters in Table 2, where the "Algorithms" column indicates which algorithms have corresponding hyperparameters in common.

H Pseudocode

Algorithm 1 DT2GS based on MAPPO

Algorithm 2 Scalable Subtask Encoder

Input: The parameters θ π En for Scalable Subtask Encoder of actor π , agent i 's observation o t i in timestep t , agent i 's subtask k t -1 i and actor RNN state h t -1 i,π in timestep t -1

Output: agent i 's subtask k t i and actor RNN state h t i,π in timestep t

1: Compute entity-embedding ( µ t i,j , σ t i,j ) , j = 1 , ..., m by Formula (2)

2: Compute observation-embedding e t i by Formula (3)

3: Compute actor RNN state h t i,π by Formula (4)

4: Compute subtask k t i with k t i ∼ Gumbel-Softmax ( h t i,π )

5: Return k t i , h t i,π

Algorithm 3 Adaptive Action Decoder

Input: The parameters θ π De for Adaptive Action Decoder of actor π , agent i 's observation o t i and subtask k t i in timestep t ,

1: Compute subtask embedding z t i with z t i = Embedding ( k t i )

Output: agent i 's action a t i in timestep t

2: Compute adaptive subtask semantics ϕ t i by Formula (7)

4: Sample action a t i from the actions' probability distribution Pr t i computed by Formula (9)

3: Compute action semantics φ t i by Formula (8)

5: Return a t i

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new neural network architecture for representing agent policies in multi-agent reinforcement learning. The architecture consists of two parts: (i) a subtask encoder that chooses the subtask to perform based on the subtask-observation history and (ii) a subtask semantics module that chooses the action based on the current observation and the output of the subtask encoder. The user specifies the number of subtasks as a hyperparameter and both modules are trained end-to-end. Experimental evaluation shows that this architecture enables SOTA performance in single-task as well as multi-task settings. Furthermore, the approach enables training policies that generalize to new scenarios in zero-shot and finetuning settings. A qualitative analysis shows that the subtasks learned can be meaningful and interpretable.\n\nSTRENGTHS:\n1. The idea of splitting the policy into two modules to decompose the overall task into subtasks is interesting and seems to work pretty well in the StarCraft environment. The ability to discover interpretable subtasks is an important benefit of the approach.\n2. Zero-shot generalization abilities are improved by this architecture. The learned policy generalizes reasonably well to new scenarios obtained by modifying the original task.\n3. The overall architecture is logical and appears easy to implement.\n\nWEAKNESSES:\n1. The work seems somewhat incremental and I am unsure if there is sufficient reason to believe the generality of the approach w.r.t. different MARL applications and scenarios. The results are purely empirical and it would strengthen the paper to consider another environment different from the StarCraft domain studied in this paper.\n2. Some design choices are not fully justified. For example, hiding the action history from the subtask encoder is mentioned as a reason to enable cross-task generalization. However, the observation history might contain information about the action history in some environments and it is unclear if hiding the action history in such a case will make a difference. In fact, I believe it is important to include an ablation in which the action history is also given to the subtask encoder to show that the proposed method is better.\n3. The architecture seems to use standard techniques and it would really improve the paper to explain the difference w.r.t. prior work and what is novel about this particular architecture---e.g., how it differs from the baselines and what parts of the architecture are borrowed from prior work.\n4. The decomposition of the task into subtasks can also be achieved using a hierarchical approach such as the one in [1]. I believe that it is important to compare the empirical performance of the approach presented in this paper with that of a hierarchical approach.\n\n[1] Yang, Jiachen, Igor Borovikov, and Hongyuan Zha. \"Hierarchical Cooperative Multi-Agent Reinforcement Learning with Skill Discovery.\" International Conference on Autonomous Agents and Multi-Agent Systems. 2020.\n\nQUESTIONS:\n1. What parts of the architecture are novel and how does it differ from prior work?\n2. Is there any prior work that tries to decompose the policy into a subtask encoder and a subtask semantics module, either in the cooperative multi-agent setting or the single-agent setting?\n3. Would it make sense to consider freezing the subtask semantics module (after training on multiple tasks) during finetuning to perform new tasks that can be accomplished using the same subtasks? This might demonstrate that the learned subtasks generalize to completely new tasks which may require performing the subtasks in a different order.\n4. Is the architecture trainable in a hierarchical way---i.e., train the subtask encoder at a higher level (grouping transitions in some way) than the subtask semantics module?\n\nLIMITATIONS:\nLimitations are not explicitly discussed and it would be good to include them---e.g., reliance on the assumption that the action space can be decomposed in a certain way (self-action and actions affecting other agents).", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on transfer learning of multi-agent reinforcement learning (MARL), by establishing generalizable sub-tasks to enable knowledge reuse. Empirical evidence underscores that the proposed algorithm demonstrates robust zero-shot generalization across a variety of tasks. Furthermore, the algorithm showcases ample transferability and surpasses contemporary methods in both multi-task and single-task problem areas.\n\nSTRENGTHS:\n(a) The design of the algorithm is presented convincingly and with remarkable clarity, providing an adequate level of detail.\n\n(b) The paper presents compelling empirical results that display the algorithm's leading-edge performance in both multi-task and single-task scenarios. The case study outlined in Section 4.3 greatly aids in illuminating its superior capabilities.\n\n(c) The exploration of transferable MARL represents a pivotal research area with substantial potential for practical applications.\n\nWEAKNESSES:\n(a) The paper seems to omit the related work section, which could be pivotal in comprehending the originality and contributions of this study.\n\n(b) As it stands, the algorithm's generalization capacity is restricted to target tasks that can be broken down into a series of task-independent subtasks derived from source tasks. If the tasks are more diverse, it would be beneficial to consider task-specific subtasks as well.\n\nQUESTIONS:\n(a) The paper seems to address the same research problem as reference [36]. It would be beneficial to clarify your unique contributions and innovations in comparison.\n\n(b) For zero-shot generalization to be applicable across multiple tasks, is it necessary for the tasks to be related or to follow the same distribution? If so, it's crucial to articulate this clearly.\n\n(c) An overview of studies on multi-task skill-based (also known as hierarchical) MARL should be provided. More broadly, it would be useful to include work on multi-agent option (also known as skill) discovery in the related works section.\n\n(d) The \"Adaptive Subtask Semantics\" section appears to heavily rely on the Action Semantics Network (ASN). Providing some background on ASN could aid in comprehension.\n\n(e) In the multi-task settings, is the sole difference among tasks the number of entities, for example, shifting from 8m to 10m?\n\n(f) Given its close relevance, it would be beneficial to include [36] as one of the baseline comparisons. (New results are not mandatory.)\n\nLIMITATIONS:\nThe limitation part is not provided.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes DT2GS (Decompose a Task inTO a series of Generalizable Subtasks) that addresses multi-agent reinforcement learning in the contexts of zero-shot generalization, transfer, and multi-task. DT2GS learns task-independent subtasks that are characterized by the effects of each agent on itself and other agents when solving the subtask. The proposed method outperforms baselines in SMAC environment in zero-shot, transfer, and multi-task scenarios.\n\nSTRENGTHS:\n**S1. Intuitive and motivating formulation of subtasks**\n\nThe concept of defining a subtask based on the influence an agent on other entities is quite interesting and intuitive. This approach effectively translates into the context of multi-agent Reinforcement Learning (RL) problems, much like how the definition of a subtask is grounded on the Markov Decision Process (MDP) dynamics in single-agent RL problems. This parallel offers a refreshing perspective on tackling multi-agent RL scenarios. Also, it is well demonstrated in Figures 5 and 6.\n\n**S2. Generalizing MARL**\n\nThe approach of expanding the scope of multi-agent Reinforcement Learning (RL) to incorporate zero-shot and multitask problems, achieved through the introduction of task-independent subtasks, is notably intriguing. Furthermore, the experiments have been designed to effectively demonstrate these problems.\n\nWEAKNESSES:\n**W1. Lack of clarity**\n\nThis paper could significantly benefit from substantial revisions to enhance its overall clarity.\n\n- Concerning the technical writing, the manuscript suffers from the frequent use of undefined notations and interchangeable deployment of different notations, which create confusion. There are discrepancies between the notations used in equations and those in figures. For instance, the subscripts n, N, and n_a are inconsistently applied, leading to ambiguity. The varying entity notations, which also diverge from those used in ASN [1], generate unnecessary confusion, despite possibly facilitating the writing process. The clarity is further compromised when it is uncertain whether terms like o_i or o^t_i refer to the raw observation or the entity (see Line 73, Line 86, and Figure 1).\n- The authors present the detailed definition of the subtasks after these terms have already been utilized in the text. To improve the flow and comprehension, it could be beneficial to relocate the definitions found on Line 143 to a position before Line 96, where they are first employed.\n- The paper also contains lengthy sentences riddled with grammatical errors (e.g., Line 138 and Line 141), detracting from the readability.\n- The description of the adaptive subtask semantics alongside the attention mechanism, specifically from Line 156 to Line 163, is notably vague and could use clearer explanations.\n\n**W2. Lack of comparisons to previous work**\n\n- The absence of a dedicated section for related work hinders the readers' ability to contextualize this study within the broader scope of the relevant literature. It is strongly suggested that the authors include a comprehensive summary of related work. This should highlight the strengths, weaknesses, differences, and improvements offered by the proposed method in comparison to other techniques in the field. Without this context, DT2GS may risk being perceived as merely a combination of ASN (entity) [1] and UPDeT (attention) [2].\n- The authors appear to have overlooked the need to reference or empirically compare their work to other related studies that also focus on subtasks in Multi-Agent Reinforcement Learning (MARL). Notably missing comparisons to RoDE [3] and LDSA [4] could provide valuable benchmarks.\n\n[1] Wang et al., “Action Semantics Network: Considering the Effects of Actions in Multiagent Systems”, ICLR 2020.\n\n[2] Hu et al., “UPDeT: Universal Multi-agent Reinforcement Learning via Policy Decoupling with Transformers,” ICLR 2021.\n\n[3] Wang et al., “RODE: Learning Roles to Decompose Multi-Agent Tasks,” ICLR 2021. \n\n[4] Yang et al., “LDSA: Learning Dynamic Subtask Assignment in Cooperative Multi-Agent Reinforcement Learning,” NeurIPS 2022.\n\n\n**Acknowledgment Following Rebuttal**\n\nThe author's rebuttal offered comprehensive revisions to the notations for enhanced clarity. Additionally, the inclusion of pseudocode aids in understanding the core concepts. I believe that implementing these proposed changes will significantly strengthen the work.\n\nQUESTIONS:\n**Q1.** The paper seems to lack a rigorous mathematical formulation that captures the definition of a subtask provided on Line 142. Could the authors clarify this aspect with a suitable formulation?\n\n**Q2.** The two-stage self-attention mechanisms utilized in the subtask semantics module do not appear to be immediately intuitive. Could the authors further clarify their ideas from Line 146 to Line 163, particularly focusing on how this structure can prompt the subtask to categorize the impact of the agent on the entities?\n\n**Q3.** It's somewhat perplexing why the MLP (Multi-Layer Perceptron) is designated as \"similarity\", considering it doesn't directly compute the similarities of the embeddings. Could the authors clarify this terminology?\n\n**Q4.** It’s unclear how the learned subtask context is actually used by the policy. A pseudocode \nof the entire process including MAPPO update would be helpful.\n\n**Q5.** The depiction of multiple encoders with a single decoder in Figure 1 is rather confusing. Given that it's a decentralized system, it seems unlikely that each agent shares all input embeddings from all encoders. Does the figure represent the encoder and decoder for the n-th agent, where only the n-th embedding is input into the decoder, excluding the other n-1 agents?\n\n**Q6.** Is the robustness of DT2GS dependent on the choice of n_k=4? Would the method's performance vary significantly with a different choice of n_k?\n\nLIMITATIONS:\nDespite the authors' affirmation in the checklist regarding limitations, neither the main manuscript nor the appendix seems to provide any explicit discussion of the potential limitations of the proposed method.\n\nA possible limitation of the proposed work could be its inability to generalize to target tasks with a larger distribution shift, especially those that necessitate the implementation of entirely novel subtasks. For instance, an agent that learns four subtasks during training may struggle when the target task requires the execution of a fifth, previously unseen subtask. Moreover, the computational and memory demands imposed by the subtask encoder and semantics module could also present significant challenges, serving as additional potential limitations of this work", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces the DT2GS framework to improve the generality of agents in Multi-Agent Reinforcement Learning (MARL) by decomposing a task into generalizable subtasks. The authors use a scalable subtask encoder to identify appropriate subtasks based on historical entity-observation pairs, instead of action-observation pairs, which enhances generalizability. With the subtask selected by the scalable subtask encoder, the paper employs an adaptive subtask semantics with a self-attention mechanism to compute adaptive action semantics that are then passed to an MLP to obtain action distribution and Q values. This adaptive action semantics contains information about the current subtask selected as well as observations from all the entities, making it sufficient for computing action and Q values. The self-attention mechanism used does not have a position embedding, which is suited to the requirement of scalability across tasks regardless of the number of entities and permutation invariance. Empirical results demonstrate that DT2GS possesses sound zero-shot generalization capability across tasks, exhibits sufficient transferability, and outperforms existing methods in both multi-task and single-task problems.\n\nSTRENGTHS:\n1. The proposed framework's structure is well suited to the problem setting, including the scalable subtask encoder and the adaptive subtask semantics.\n2. Experimental results are promising, demonstrating improved performance compared to baseline methods, as well as providing insightful analysis on subtask percentage.\n\nWEAKNESSES:\n1. A more detailed introduction of related work could be helpful for readers if space allows, such as the network design or methodology of ASN.\n2. In Figure 1, the right part uses \"Env Cognition\" and \"Cognition Encoder: MLP\". Adding the word \"cognition\" to appropriate locations in the text around Equation 2 would make Figure 1 match the text better.\n\nQUESTIONS:\nFigure 4 only reports the performance of each algorithm trained on the source task but tested on the target task. What is the performance of algorithms on the source task? If one algorithm performs poorly even on the source task, then it makes little sense to test it on a different target task.\n\nLIMITATIONS:\nThe authors did not explicitly discuss the limitations of their work in the paper. However, no major limitations or potential negative societal impacts have been identified in this review.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}