# A Multitask-Based Transfer Framework for Cooperative Multi-Agent Reinforcement Learning
## 4 Methodology
### Handling Gradient Conflict in Multitask Training  
We focus on the optimization conflict problem in multitask learning as shown in Figure 2. The gradient size and direction of different tasks may be different, which may lead to gradient conflict and hinder model learning [42; 60]. The degree of gradient conflict is usually measured by gradient cosine similarity [61]. We aim to reduce the conflict in the optimization process by processing the gradient in multitask training. Specifically, we consider a multitask training optimization process with N tasks, i.e., \(\{\mathcal{T}_{1},\mathcal{T}_{2},\dots,\mathcal{T}_{n}\}\). The gradients of the loss functions corresponding to each task are represented by \(\{\mathbf{g}_{1},\mathbf{g}_{2},\dots,\mathbf{g}_{n}\}\). There is a gradient \(\mathbf{g}_{o}\), and we assume the gradient \(\mathbf{g}_{i}\) of task \(\mathcal{T}_{i}\). These two gradients are conflicting if \(\mathbf{g}_{i}^{\top}\mathbf{g}_{o}\) is negative [62]. Therefore, in order to update the gradient without conflict, it is necessary to ensure that \([\mathbf{g}_{1},\mathbf{g}_{2},\dots,\mathbf{g}_{n}]^{\top}\mathbf{g}_{o}\) are positive. The above conditions for conflict-free gradient updates can be transformed into \(\mathbf{g}_{o}=[\mathbf{g}_{1},\mathbf{g}_{2},\dots,\mathbf{g}_{n}]^{-\top} \mathbf{w}\), where \(\mathbf{w}=[w_{1},w_{2},\dots,w_{n}]\) is a vector with n positive components. Here, \(\mathcal{M}^{-\top}\) represents the pseudoinverse of the transposed matrix \(\mathcal{M}^{\top}\). A positive \(\mathbf{w}\) vector guarantees a conflict-free update direction for all losses, it is also necessary to determine the specific direction of \(\mathbf{g}_{o}\). Since the value of \(w_{i}\) influences the precise orientation of \(\mathbf{g}_{o}\), it is essential to regulate the specific value of \(w_{i}\) accordingly. We reformulate \(\mathbf{g}_{o}\) as \(\mathbf{g}_{o}=k[\mathcal{U}(\mathbf{g}_{1}),\mathcal{U}(\mathbf{g}_{2}), \dots,\mathcal{U}(\mathbf{g}_{n})]^{-\top}\hat{\mathbf{w}}\), where \(\mathcal{U}(\mathbf{g}_{i})=\mathbf{g}_{i}/\left(|\mathbf{g}_{i}|+\epsilon\right)\) is a normalization operator and \(k>0\). Now, \(k\) controls the length of \(\mathbf{g}_{o}\) and the ratio of \(\hat{\mathbf{w}}\)'s components corresponds to the ratio of \(\mathbf{g}\)'s projections onto each task-specific \(\mathbf{g}_{i}\). \(\hat{\mathbf{w}}\) can be called the direction weight. The projection length of \(\mathbf{g}_{o}\) on each task-specific gradient serves as an effective \"learning rate\" for each task. However, setting a fixed k value is not always optimal during the optimization process, as the length of the gradient vector g should be adaptively adjusted according to different optimization scenarios. Specifically, when all gradients related to the task nearly point in the same direction, it indicates the presence of a favorable optimization direction, and the length of \(\mathbf{g}_{o}\) should be increased to enhance optimization in this direction. Conversely, if the gradients for specific tasks are close to each other, the length of \(\mathbf{g}_{o}\) should be decreased to avoid over-optimization. Addressing this issue, we propose an adaptive strategy where the length of \(\mathbf{g}_{o}\) is adjusted to the sum of the projected lengths of each task-specific gradient, thereby dynamically controlling the length of \(\mathbf{g}_{o}\). The above process is summarized as follows:  
\[\mathbf{g}_{o}=\mathcal{G}(\mathbf{g}_{1},\mathbf{g}_{2},\dots, \mathbf{g}_{n}):=\left(\sum_{i=1}^{n}\mathbf{g}_{i}^{\top}\mathbf{g}_{a}\right) \mathbf{g}_{a} \tag{3}\] \[\mathbf{g}_{a}=\mathcal{U}\left[\mathcal{U}(\mathbf{g}_{1}), \mathcal{U}(\mathbf{g}_{2}),\dots,\mathcal{U}(\mathbf{g}_{n})]^{-\top}\mathbf{ E}_{n}\right] \tag{4}\]  
where \(\mathbf{E}_{n}\) is a unit vector with n components.