# A Multitask-Based Transfer Framework for Cooperative Multi-Agent Reinforcement Learning
## 4 Methodology
### Knowledge Transfer Based on Multitask Training  
As illustrated in Figure 1, our framework encompasses two phases: the knowledge extraction phase and the knowledge transfer phase. **Knowledge Extraction**: The knowledge extraction component adheres to a conventional multitask training architecture, comprising task-specific observation embedding networks and decision networks, as well as a shared knowledge extraction network for all tasks. Suppose we have N tasks \([\mathbf{t_{1}},\mathbf{t_{2}},\dots,\mathbf{t_{n}}]\). Consequently, we will have N observation embedding networks, N action mapping networks, and one knowledge extraction net  
Figure 1: Overview of the architecture of our proposed framework. work. The architecture for each task-specific observation embedding network is uniform, consisting of a two-layer neural network with an initial fully connected layer followed by an LSTM layer. The input dimensions of each task's observation embedding network vary due to the distinct state space dimensions across different tasks. The observation embedding network takes the task's state \(s\) as input and outputs a fixed 256-dimensional embedding. Subsequently, a shared knowledge extraction network, which is a two-layer fully connected neural network, processes the output of the observation embedding networks to produce a fixed-dimensional embedding. Similar to the observation embedding networks, the decision networks for all tasks share an identical architecture, consisting of a two-layer fully connected neural network that takes the output of the shared knowledge extraction network as input and produces the final output. The final layer of the decision network is adjusted according to the action space dimensions of the respective task, as each task's action space dimensionality may differ. For each task, we employ the IPPO algorithm for training. IPPO is a multi-agent reinforcement learning algorithm that extends the widely used proximal policy optimization (PPO) algorithm to multi-agent settings. Unlike centralized approaches that rely on global observations or joint action spaces, IPPO adopts a decentralized training paradigm, where each agent independently optimizes its own policy while interacting with the environment and other agents. This independence simplifies training and scales efficiently to scenarios with a large number of agents. IPPO assumes a partially observable Markov game, where each agent \(i\) has its own observation space \(\mathcal{O}_{i}\), action space \(\mathcal{A}_{i}\), and policy \(\pi_{\theta_{i}}(a_{i}|o_{i})\) parameterized by \(\theta_{i}\). The goal of each agent is to maximize its cumulative discounted reward:  
\[J_{i}(\theta_{i})=\mathbb{E}_{\tau\sim\pi_{\theta_{i}}}\left[\sum_{t=0}^{T} \gamma^{t}r_{i}^{t}\right] \tag{1}\]  
where \(\tau\) denotes the trajectory, \(\gamma\) is the discount factor, and \(r_{i}^{t}\) is the reward received by agent \(i\) at time step \(t\). Similar to PPO, IPPO uses a clipped surrogate objective to update each agent's policy. The objective function for agent \(i\) is given by:  
\[L_{i}(\theta_{i})=\mathbb{E}_{t}\Big{[}\min\Big{(}r_{t}(\theta_{i})\hat{A}_{t} ^{i},\text{clip}(r_{t}(\theta_{i}),1-\epsilon,1+\epsilon)\hat{A}_{t}^{i}\Big{)} \Big{]} \tag{2}\]  
where \(r_{t}(\theta_{i})=\frac{\pi_{\theta_{i}}(a_{i}^{i}|\theta_{i}^{i})}{\pi_{ \theta_{i}^{old}(a_{i}^{i}|\theta_{i}^{i})}}\) is the probability ratio between the new and old policies, \(\hat{A}_{t}^{i}\) is the advantage estimate for agent \(i\), \(\epsilon\) is a hyperparameter controlling the clipping range. Within each episode, tasks are trained in a round-robin fashion until all tasks have converged or the predefined maximum number of training steps is reached. The shared knowledge extraction network obtained upon completion of training is regarded as the common knowledge extracted during multitask training. This network is then transferred to new tasks to enhance the collaborative performance of multi-agent teams.