SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this section, we analyzed the practical semantics of subtasks based on the marine-series tasks. We first trained a DT2GS model from scratch on the 8m task and then deployed it to 8m and  $10\mathrm{m\_vs\_11m}$ , as shown in Figure 5 left. In the 8m and  $10\mathrm{m\_vs\_11m}$  tasks, the DT2GS agents initially utilized subtask_1 to scatter the formation and advance towards the enemy at  $t = 1$ . When the enemies reached the attack range of DT2GS agents at  $t = 6$ , most of these agents selected subtask_2 to focus fire on enemies while also paying attention to their own health in order to move back when in danger, thereby avoiding the decrease in firepower caused by personnel reduction. And other agents chose subtask_4 to slightly adjust their relative position with the enemies to seek a better position for firepower output. At  $t = 13$ , some agents sensed that their allies' health was too low while their own health was safe enough. These agents switched subtask from subtask_2 to subtask_3, which means charging. And the specific behavior was to attack while moving closer to the enemies' position to attract hatred so as to avoid the decrease in firepower caused by the death of allies. At the end of the battle (after  $t = 21$ ), all agents changed their subtasks to subtask_3 to concentrate their firepower to end the battle. We observed that under the subtask of charge, even if an agent was close to death, it would not move back to disperse hatred as overall withdrawal would lead to failed combat.
Furthermore, we conducted zero-shot generalization experiments on additional marine tasks, and the corresponding change process of subtask percentage is shown in Figure 6. We observed that the change process of each subtask's percentage in the source task (8m) is quite similar to that in the target tasks (10m_vs_11m, 5m_vs_6m, 8m_vs_9m, 25m, and 27m_vs_30m). This result demonstrates that the subtasks decomposed by DT2GS from the source task (8m) are task-independent across tasks and
Figure 5: Visualization of the practical semantics of subtasks (left) and the change process of subtask percentage (right) in  $8\mathrm{m}$  (up) and  $10\mathrm{m\_vs\_11m}$  (down). The results for both tasks are obtained from the source model's evaluation on the corresponding task, where the source model is acquired by learning from scratch in task  $8\mathrm{m}$ .
therefore effective for the target tasks. Both results from Figure 5 and Figure 6 further suggest that DT2GS policy has sound zero-shot generalization capability by decomposing the task into a series of task-independent subtasks.
Figure 6: Changes of subtask percentage when deploying the source model, which is learned from scratch on  $8\mathrm{m}$ , to 6 marine tasks. The subtask percentage is calculated as  $p(k) = \frac{N_k}{N_a}$ , where  $p(k)$  is the percentage of subtask  $k$ ,  $N_{k}$  is the number of agents that select subtask  $k$ , and  $N_{a}$  is the total number of agents.
4.4 Transferability across Tasks
Figure 7 (a) illustrates four scenarios that we designed to evaluate the transferability of DT2GS policy. In all four scenarios, the relationships from the source task to the target task are extrapolation, including an increase in entity types (3s5z_vs_3s6z  $\rightarrow$  1c3s5z), an increase in the number of enemies (3s5z  $\rightarrow$  3s5z_vs_3s6z and 3s_vs_4z  $\rightarrow$  3s_vs_5z), as well as an increase in the total scale of entities (8m_vs_9m  $\rightarrow$  27m_vs_30m).
As we can see, DT2GS_finetune demonstrated significantly efficient convergence as well as higher and more stable asymptotic performance compared with UPDeT_finetune and ASN_G_finetune in all transfer scenarios. Notably, in the scenario where the source task was 3s5z (easy) and the target task was 3s5z_vs_3s6z (superhard), DT2GS_finetune achieved optimal performance with only 6400 steps interaction with environment, whereas learning from scratch required at least 6 million steps. Furthermore, compared to other baselines of learning from scratch, DT2GS_finetune accelerated
the model's convergence on target tasks by an average of  $100 \times$ . In conclusion, DT2GS_finetune exhibited better performance than baselines on the following evaluation metrics [28, 4]: jumpstart, time to threshold and asymptotic performance, demonstrating significantly sufficient transferability.
Figure 7: (a) Comparison of transferability between DT2GS and baselines, where baselines include transfer of UPDeT (UPDeT_finetune), ASN_G (ASN_G_finetune) and learning from scratch of DT2GS, UPDeT, ASN, MAPPO. (b) Comparison of performance on Single-Task between DT2GS and baselines, including UPDeT, ASN, MAPPO, HSD, RODE, ROMA and LDSA, on 3 superhard tasks (3s5z_vs_3s6z, 6h_vs_8z, corridor) and 1 hard task (5m_vs_6m).
(a) Transferability
(b) Single-Task
4.5 Performance on Multi-task and Single-task
In this section, we first designed 2 multi-task problems, including the marine-series tasks ( $\{3\mathrm{m}, 8\mathrm{m}, 8\mathrm{m\_vs\_9m}, 10\mathrm{m\_vs\_11m}\}$ ) and stalker_zealot-series tasks ( $\{2\mathrm{s}3\mathrm{z}, 3\mathrm{s}5\mathrm{z}, 3\mathrm{s}5\mathrm{z\_vs\_3s6z}\}$ ), to demonstrate the representational capacity of DT2GS. In each multi-task problem, the policy interacted synchronously with multiple tasks that make up this multi-task to collect data, which was shuffled and used to update the policy. As shown in Figure 8, DT2GS exhibited better learning efficiency and more stable asymptotic performance compared with UPDeT and ASN_G.
Figure 8: Comparison of performance on multi-task problems between DT2GS and baselines, including UPDeT and ASN_G.
Subsequently, we also compared DT2GS with baselines on single-task scenarios. We selected four representative scenarios, including superhard / hard tasks, as shown in Figure 7 (b), while performance on other scenarios is presented in the Appendix B. Our evaluation metrics included learning efficiency and final asymptotic performance. Compared to MAPPO, DT2GS significantly improved both metrics, particularly in superhard tasks such as  $3s5z\_ vs\_ 3s6z$  and  $6h\_ vs\_ 8z$ . In addition, compared to other baselines based on MAPPO, including UPDeT and ASN, DT2GS consistently outperformed them in terms of learning efficiency and asymptotic performance, on average.
Model generalization has emerged as a promising approach to reduce training costs. In this paper, we proposed DT2GS, an effective approach for addressing the problem of model generalization across tasks in the MARL field. Our insight is that task-independent subtasks exist across tasks, making it possible to generalize the model across tasks. Based on this insight, we assumed that the model can be generalized to target tasks if it can decompose task-independent subtasks from source tasks. The challenge then becomes ensuring that the subtasks we decompose from the source task are truly task-independent. Regarding this issue, we proposed two properties that enable task-independence of subtasks: (1) avoiding overfitting to the source task, (2) maintaining consistent yet scalable semantics across tasks. Then we proposed DT2GS to endow the subtasks with these two properties by introducing the scalable subtask encoder and the adaptive subtask semantic module, respectively. Empirical results demonstrated that DT2GS can decompose tasks into a series of generalizable subtasks, leading to a generalizable MARL policy. Nevertheless, it would be beneficial to consider task-specific subtasks as well when there is a significant distribution shift between source and target tasks. In our future work, we will focus on expanding the generalization scenarios of DT2GS to address this limitation.
This work is partially supported by the NSF of China (under Grants 61925208, U22A2028, 62222214, 62002338, 62102399, U19B2019, 92364202), CAS Project for Young Scientists in Basic Research (YSBR-029), Youth Innovation Promotion Association CAS and Xplore Prize.
[1] Akshat Agarwal, Sumit Kumar, and Katia Sycara. Learning transferable cooperative behavior in multi-agent teams. arXiv preprint arXiv:1906.01202, 2019.
[2] Georgios Boutsioukis, Ioannis Partalas, and Ioannis Vlahavas. Transfer learning in multi-agent reinforcement learning domains. In *Recent Advances in Reinforcement Learning: 9th European Workshop*, EWRL 2011, Athens, Greece, September 9-11, 2011, Revised Selected Papers 9, pages 249-260. Springer, 2012.
[3] Kyunghyun Cho, Bart Van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078, 2014.
[4] Felipe Leno Da Silva and Anna Helena Reali Costa. A survey on transfer learning for multiagent reinforcement learning systems. Journal of Artificial Intelligence Research, 64:645-703, 2019.
[5] Sabre Didi and Geoff Nitschke. Multi-agent behavior-based policy transfer. In Applications of Evolutionary Computation: 19th European Conference, EvoApplications 2016, Porto, Portugal, March 30–April 1, 2016, Proceedings, Part II 19, pages 181–197. Springer, 2016.
[6] Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
[7] Qingxu Fu, Tenghai Qiu, Jianqiang Yi, Zhiqiang Pu, and Shiguang Wu. Concentration network for reinforcement learning of large-scale multi-agent systems. arXiv preprint arXiv:2203.06416, 2022.
[8] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735-1780, 1997.
[9] Siyi Hu, Fengda Zhu, Xiaojun Chang, and Xiaodan Liang. Updet: Universal multi-agent reinforcement learning via policy decoupling with transformers. arXiv preprint arXiv:2101.08001, 2021.
Transfer learning across tasks in online MARL Transfer learning across tasks is a promising approach to accelerate the convergence of models, especially in large-scale MAS. The first method of transfer learning in online MARL domains is BITER [2], which uses the joint policy learned in the source task to bias the initial policy of agents in the target tasks. After more than a decade of development, some methods have been proposed in this field. In this regard, we classify these methods into two categories based on their characteristics, namely Network-design-based methods and Task-embedding-based methods.
The Network-design-based methods implicitly reuse knowledge from source tasks by constructing universal model structure across tasks using Population Invariant Structure such as Transformer [9] or GNN [1]. Since the existence of varying state/observation/action space across tasks, the Network-design-based methods are the foundation of all other methods. And the Task-embedding-based methods are knowledge transfer by calculating task similarity using learned task embeddings that capture task dynamics [2, 5, 14, 23, 19]. For example, MATTAR [19] models the target task representation as a weighted sum of a series of source task representations and decodes the task representation to the policy weight that is suitable for the task by utilizing a hypernetwork. And MATE [23] models task representation by reconstructing the task dynamics, so as to achieve task differentiation and fast adaptation of policy on tasks.
Previous works focused on multi-agent transfer learning based on network-design or task-embedding, but lacked of efficient knowledge reuse, leading to limited generalization capability. Our work (DT2GS) leverages knowledge reuse by ensuring semantic consistency and scalability between diverse tasks and preventing over-fitting of source tasks, which greatly improves transferability and even zero-shot generalization capability.
Transfer learning across tasks in other paradigms There are some methods dedicated to transfer learning across tasks in MARL belonging to other paradigms. For example, ODIS [41], belonging to the offline RL paradigm, discovers generalizable skills from multi-task offline data to enhance the model's generalization capability to unseen tasks. Besides, EPC [15] and DyMA-CL [34] fulfill knowledge transfer across tasks by way of Curriculum Learning.
Skills/Options/Roles/Subtasks in MARL There are some works in MARL focusing on concepts like skills/options/roles, which are similar to subtasks studied in our method. LDSA [38] learns dynamic subtask assignments and constructs different policies for different subtasks, so enabling agents to possess specific abilities to handle different subtasks. ROMA [31] introduces the role concept into MARL in the form of Gaussian distribution and assigns certain subtasks to agents with similar roles. RODE [32] further studies how to efficiently discover the roles set by decomposing joint action spaces into restricted role action spaces. HSD [37] is proposed to discover complementary and useful skills for cooperative team play in MARL.
ASN [33] && UPDeT [9] For model generalization in MAS, a generalizable model structure is necessary for addressing the problem of varying state/observation/action space  $(S/O/A)$  across tasks. Here we define the agents controlled by MARL policy and agents built-in tasks as entities. As demonstrated in ASN [33], an agent's observation  $o_i$  can be constructed as a concatenation of  $m$  entity-observations:  $o_i = [o_{i,1}, o_{i,2}, \dots, o_{i,m}]$ , where  $o_{i,1}$  is the observation of agent  $i$  on itself and environment, and the rest are the observations of agent  $i$  on other  $m - 1$  entities. Additionally, action space  $\mathcal{A}$  can be decomposed into two components:  $\mathcal{A}^{self}$ , which consists of actions affecting the agent itself or the environment, and  $\mathcal{A}^{interactive}$ , which contains actions that directly interact with other entities. This alignment between entity-observations and actions, which is referred to as action semantics, forms the foundation for computing the value or probability of an action based on its aligning entity-observation, leading to a generalizable model structure across tasks.
Thereafter, by combining ASN with Transformer [29], UPDeT [9] develops a type of population-invariant network (PIN) to further improve the model's generalization capability. Specifically, entity-observations are taken as queries, keys, and values to derive an attention output, as shown in Formula (10) and Formula (11).
$$
Q _ {i} ^ {t} = W _ {Q} \left[ o _ {i, 1} ^ {t}, o _ {i, 2} ^ {t}, \dots , o _ {i, m} ^ {t} \right], K _ {i} ^ {t} = W _ {K} \left[ o _ {i, 1} ^ {t}, o _ {i, 2} ^ {t}, \dots , o _ {i, m} ^ {t} \right], V _ {i} ^ {t} = W _ {V} \left[ o _ {i, 1} ^ {t}, o _ {i, 2} ^ {t}, \dots , o _ {i, m} ^ {t} \right] \tag {10}
$$
$$
\varphi_ {i} ^ {t} = \left[ \varphi_ {i, 1} ^ {t}, \dots , \varphi_ {i, m} ^ {t} \right] = \operatorname {s o f t m a x} \left(\frac {Q _ {i} ^ {t} K _ {i} ^ {t T}}{\sqrt {d _ {K}}}\right) V _ {i} ^ {t} \tag {11}
$$
Subsequently, the attention output  $\varphi_{i,1}^{t}$  and  $\varphi_{i,j}^{t}, j = 2,\dots,m$  are utilized to obtain the Q-value or probability of actions in  $\mathcal{A}^{self}$  and  $\mathcal{A}^{interactive}$ , respectively. As shown in ODIS [41], UPDeT [9] and [1], the pipeline of PIN has been regarded as a foundational framework for the model's generalization across tasks in MARL. Therefore, the PIN is also employed in DT2GS.
B Asymptotic Performance on Single-Task problems
Figure 9: Comparison of DT2GS performance with baselines, including UPDeT, ASN, and MAPPO, on Single-Task Problems.
C Consistent yet Scalable semantics of subtask
DT2GS endows subtasks with consistent yet scalable semantics across tasks. For instance, the semantics of subtask  $k$  could involve tempting 2 enemies in the source task while 4 enemies in the target task. The term "consistency" refers to  $k$  tempting enemies in both the source and target tasks, while "scalability" refers to the changed impact on entities. Figure 10 illustrates the subtask semantics, which are represented by the subtask's attention on all entities. We observed that DT2GS can adapt subtask semantics to different task contexts effectively, reflecting its scalability on subtask semantics across tasks. In our experiments, agents paid attention solely to allies as they employed the subtask "disperse" to line up during the t_1 period. During the t_2 period, agents initially paid attention on both allies and enemies as they utilized the subtask "hit and run" to focus fire on enemies while monitoring their own health and allies' position. Later in t_2, agents shifted their focus to enemies as they employed the subtask "charge" to concentrate their firepower to end the battle. The last period, t_3, signifies the agents' demise.
Figure 10: Visualization of the subtask semantics, which represented by the subtask's attention on all entities. We trained a model on the source task  $8\mathrm{m}$  and evaluated its performance on 6 different marine tasks. To visualize the semantics of subtasks assigned to the agent during an episode, we plotted the subtask attention on all entities over time. The resulting visualization is shown above, where each cell represents the attention weight of a subtask-entity pair at a specific timestep. The horizontal axis denotes the entities present in the task, ordered from allies to enemies from left to right, while the vertical axis represents the chronological order of timesteps in the episode from top to bottom.
D Generalization on MPE
Figure 11: The figure shows a comparison of zero-shot generalization capability between DT2GS, UPDeT, and ASN_G on the physical deception task (Spread) in the multi-agent particle world environments (MPE) [16]. The horizontal axis represents the source task  $\rightarrow$  the target task. For example, "3→4" indicates that the source task is set by 3 agents and 3 landmarks, while the target task is set by 4 agents and 4 landmarks. The vertical axis represents the average episode reward when deploying the source model to the target task without any finetue. The red, green, and purple histograms correspond to DT2GS, UPDeT, and ASN_G, respectively. As we can see, DT2GS significantly outperforms UPDeT and ASN_G in terms of zero-shot generalization capability, achieving an average episode reward surpass of about 17.05 and 44.32, respectively
E Ablation Study
To investigate the effect of the number of subtasks  $n_k$  on the model's zero-shot generalization capability, we conducted an ablation study, as shown in Figure 12. Our results indicate that appropriately increasing the number of subtasks can improve the model's zero-shot generalization capability, but setting  $n_k = 6$  leads to degradation due to the limited number of generalizable subtasks across tasks. Additionally, we found that increasing  $n_k$  from 4 to 5 does not significantly improve performance, particularly in the zero-shot generalization scenario  $8\mathrm{m} \rightarrow 25\mathrm{m}$ . To balance efficiency and effectiveness, we set  $n_k = 4$  for all experiments presented in this paper.
Figure 12: Ablation study on the impact of subtask number  $n_k$  on the model's zero-shot generalization capability.
F List of symbol definitions
Table 1: List of symbol definitions
Symbol	Definitions
o_i	the observation of agent i
o_{i,j}	the entity-observation of agent i on entity j
{o_{i,j}^t}	all entity-observations of agent at time t
{o_{self}}	the entity-observation of agent on itself and environment
{o_allyj}	the entity-observation of agent on its jth ally
{o_emyj}	the entity-observation of agent on its jth enemy
a_i^t	the action of agent i at time t
k_i^t	the subtask assigned to agent i at time t
h_i^t	the history latent embedding of agent i at time t
N(μi,j^t, σi,j^t)	the embedding Gaussian Distribution of entity-observation of agent i on entity j at time t
N(μi^t, σi^t)	the embedding Gaussian Distribution of observation of agent i at time t
e_i^t	the observation embedding of agent i at time t
φ_i^t	the adaptive semantics of subtask assigned to agent i at time t
φ_i^t	the semantics of all actions for agent i at time t
G Hyperparameters
Table 2: List of Hyperparameters
Hyperparameters	Value	Algorithms
hidden layer dimension of DT2GS's Encoder	8	DT2GS
MLP's hidden layer dimension	64	DT2GS, UPDeT, ASN, ASN_G, MAPPO
attention's hidden layer dimension	64	DT2GS, UPDeT, ASN_G
attention's heads	3	DT2GS, UPDeT, ASN_G
number of subtasks	4	DT2GS
optimizer	Adam	DT2GS, UPDeT, ASN, ASN_G, MAPPO
learning rate of actor and critic	0.0005	DT2GS, UPDeT, ASN, ASN_G, MAPPO
To ensure consistency and comparability across our experiments, we aimed to share hyperparameters among algorithms wherever possible. We provide a list of the shared hyperparameters in Table 2, where the "Algorithms" column indicates which algorithms have corresponding hyperparameters in common.
H Pseudocode
Algorithm 1 DT2GS based on MAPPO
Input: The parameters  $\theta_{En}^{\pi}$  for Scalable Subtask Encoder of actor  $\pi$ ; the parameters  $\theta_{De}^{\pi}$  for Adaptive Action Decoder of actor  $\pi$ ; the parameters  $\theta^{V}$  for critic  $V$ ; the number  $n_k$  for subtasks  
Output: The Scalable Subtask Encoder  $\pi_{En}$  and Adaptive Action Decoder  $\pi_{De}$  for actor  $\pi$ ; the critic  $V$   
1: Initialize  $\theta_{En}^{\pi}, \theta_{De}^{\pi}, \theta^{V}, n_k$   
2: Initialize the total timesteps stepmax interaction with environment; the total timesteps  $T$  of an episode; the number of episodes batch_size for each actor/critic update  
3: Initialize step  $\leftarrow 0$   
4: while step  $\leq$  stepmax do  
5: set data buffer D = {}  
6: for idx = 1 to batch_size do  
7:  $\tau = []$  empty list  
8: initialize actor RNN states  $h_{1,\pi}^{0}, \ldots, h_{n,\pi}^{0}$  for each agent  
9: initialize critic RNN states  $h_{1,V}^{0}, \ldots, h_{n,V}^{0}$  for each agent  
10: initialize substasks  $k_{1}^{0}, \ldots, k_{n}^{0}$  for each agent ▷ Initialize substasks with  $n_k$ -dim zero vector  
11: for timestep  $t = 1$  to  $T$  do  
12: for all agents  $i$  do  
13:  $k_{i}^{t}, h_{i,\pi}^{t} = \pi_{En}(o_{i}^{t}, k_{i}^{t-1}, h_{i,\pi}^{t-1}; \theta_{En}^{\pi})$  ▷ Call for Scalable Subtask Encoder Module 2  
14:  $a_{i}^{t} = \pi_{De}(o_{i}^{t}, k_{i}^{t}; \theta_{De}^{\pi})$  ▷ Call for Adaptive Action Decoder Module 3  
15:  $v_{i}^{t}, h_{i,V}^{t} = V(s_{i}^{t}, h_{i,V}^{t-1}; \theta^{V})$   
16: end for  
17: Execute actions  $a^{t}$ , observe  $r^{t}, s^{t+1}, o^{t+1}$   
18:  $\tau + = [s^{t}, o^{t}, h_{\pi}^{t}, h_{V}^{t}, k^{t}, a^{t}, r^{t}, s^{t+1}, o^{t+1}]$   
19: end for  
20: step+=T  
21: Compute advantage estimate  $\hat{A}$  via GAE on  $\tau$   
22: Compute reward-to-go  $\hat{R}$  on  $\tau$   
23: Split trajectory  $\tau$  into chunks of length L  
24: for l=0,1,...,T/L do  
25:  $D = D \cup (\tau[l:l+T], \hat{A}[l:l+L], \hat{R}[l:l+L])$   
26: end for  
27: end for  
28: for mini-batch  $k = 1, \ldots, K$  do  
29: b  $\leftarrow$  random mini-batch from  $D$  with all agent data  
30: for each data chunk  $c$  in the mini-batch  $b$  do  
31: update RNN hidden states for  $\pi_{En}$  and  $V$  from first hidden state in data chunk  
32: end for  
33: end for  
34: Adam update  $\theta_{En}^{\pi}, \theta_{De}^{\pi}$  on actor loss with data  $b$   
35: Adam update  $\theta^{V}$  on critic loss with data  $b$   
36: end while  
37: Return  $\pi_{En}, \pi_{De}, V$
Algorithm 2 Scalable Subtask Encoder
Input: The parameters  $\theta_{En}^{\pi}$  for Scalable Subtask Encoder of actor  $\pi$ , agent  $i$ 's observation  $o_i^t$  in timestep  $t$ , agent  $i$ 's subtask  $k_i^{t-1}$  and actor RNN state  $h_{i,\pi}^{t-1}$  in timestep  $t-1$
Output: agent  $i$ 's subtask  $k_{i}^{t}$  and actor RNN state  $h_{i,\pi}^{t}$  in timestep  $t$
1: Compute entity-embedding  $(\mu_{i,j}^{t},\sigma_{i,j}^{t}),j = 1,\dots,m$  by Formula (2)
2: Compute observation-embedding  $e_i^t$  by Formula (3)
3: Compute actor RNN state  $h_{i,\pi}^{t}$  by Formula (4)
4: Compute subtask  $k_{i}^{t}$  with  $k_{i}^{t} \sim$  Gumbel-Softmax  $(h_{i,\pi}^{t})$
5: Return  $k_{i}^{t}, h_{i,\pi}^{t}$
Algorithm 3 Adaptive Action Decoder