# A Multitask-Based Transfer Framework for Cooperative Multi-Agent Reinforcement Learning
###### Abstract  
Multi-agent reinforcement learning (MARL) has proven to be effective and promising in team collaboration tasks. Knowledge transfer in MARL has also received increasing attention. Compared to knowledge transfer in single-agent tasks, knowledge transfer in multi-agent tasks is more complex due to the need to account for coordination among agents. However, existing knowledge transfer-based methods only focus on strategies or agent-level knowledge in a single task, and the transfer of knowledge in such a specific task to new and different types of tasks is likely to fail. In this paper, we propose a multitask-based training framework termed MTT in cooperative MARL, which aims to learn shared collaborative knowledge across multiple tasks simultaneously and then apply it to solve other related tasks. However, models obtained from multitask learning may fail on other tasks because the gradients from different tasks may conflict with each other. To obtain a model with shared knowledge, we provide conflict-free updates by ensuring a positive dot product between the final update and the gradient of each specific task. It also maintains a consistent optimization rate for all tasks. Experiments conducted in two popular environments, StarCraft II Multi-Agent Challenge and Google Research Football, demonstrate that our method outperforms the baselines, significantly improving the efficiency of team collaboration. c 28 January 202515 February 202517 February 2 algorithms. Previous studies have explored various dimensions, particularly focusing on leveraging knowledge from other tasks to facilitate learning for new tasks, a strategy commonly referred to as transfer learning. Take StarCraft II game as an example, there are many symmetric tasks, such as \(3m\), \(8m\), \(3s3z\), and \(3s5z\). In the \(3m\) and \(8m\), the number of agents differs, but the types of agents remain the same. When facing the same type of opponent, these two tasks are likely to share cooperative patterns or knowledge. In the above case, it is very suitable to transfer the knowledge of the learned tasks to new similar tasks to improve learning efficiency and team collaboration. For example, [PERSON] et al. [12] proposed a multi-agent common knowledge reinforcement learning method for collaborative tasks in StarCraft II. However, this approach assumes that the agents share common observations as common knowledge, whereas in practice, the agents may not have access to the same observations. [PERSON] et al. [13] utilized decision-making knowledge to enhance team collaboration performance, yet this method only supports value-based algorithms. [PERSON] et al. [14] introduced a knowledge transfer method with an intermediate task for confrontation decision-making. In the field of MARL, the majority of transfer learning research focuses on agent-centered knowledge, with only a few studies involving transfer at the policy level. For instance, [PERSON] et al. [15] proposed a transfer learning method based on an attention mechanism. Nonetheless, this approach is grounded in the transfer of knowledge from a single task and necessitates a high degree of similarity between the source and target tasks for effective transfer. Furthermore, existing works on transfer learning in MARL ignore cross-task knowledge. In this paper, we propose a multitask-based transfer framework aiming to improve team collaboration performance in multi-agent reinforcement learning. Different from other approaches that reuse the policy of a trained task or the coordination knowledge of multiple agents, our method consists of two stages. First, we extract the shared cooperation knowledge in multiple tasks through multitask training and then transfer it to the new task. In our framework, each task is equipped with an observation embedding network and a decision network, and a shared knowledge extraction network is set between them. It is expected that after multitask training, this shared knowledge extraction network can extract the shared collaborative knowledge across multiple tasks.