## 4 Method
### Cross-Group Prediction Loss  
In a multi-agent system (MAS), a cooperative joint policy is usually approached by maximizing the mutual information between agents. However, calculating mutual information must be taken between two groups and the overall computational complexity is proportional to the combination number of two groups out of K groups \(\Omega(C_{K}^{2})\).  
Introducing the GAS provides another available way to learn a cooperative policy, which is learning an agent's policy while also predicting other agents' policies. \"Learning with predicting\" is a natural way for humans to practice and perform cooperation, and is useful to develop tacit team policy. Instead of directly optimizing the mutual information  
Figure 1: An illustration of the usage of GAS.  
between the trajectories \(\tau_{i}\) and \(\tau_{j}\), following the \"learning with predicting\" scheme, we introduce the _Cross-Group Prediction loss (CGP loss)_.  
The core innovation is the _predictive available action mask_\(AM^{pred}\). Since the action policy is generated through the masking operation \(\theta_{i}=\theta^{U}\odot AM_{i}\), we can generate the action policy of other physically heterogeneous agents by applying their mask to the agent's GAS policy: \(\theta_{i}^{pred}=\hat{\theta}^{U}\odot AM_{i}^{pred}\). Specifically, we add an independent network branch \(\psi_{i}\) using the trajectory of the agent \(\tau_{i}\) to calculate the \(\hat{\theta}^{U}\) and the \(\theta_{i}^{pred}\). A common implementing method for encoding trajectory is to use the hidden state of the _gated recurrent unit_ (GRU) [58]\(h_{i}\). Because the GRU takes \(o_{i}^{l}\) and \(a_{i}^{l}\) as the input recursively for all time-steps \(t\), we assume that \(h_{i}\) can appropriately represent \(\tau_{i}\). The prediction process can be considered as simulating the \(\theta_{j}\) of another heterogeneous agent \(j\) with the \(\theta_{i}^{pred}\) of agent \(i\). The process is capable of enhancing the mutual information between \(\theta_{j}\) and \(\theta_{i}^{pred}\) and therefore the mutual information between \(\tau_{j}\) and \(\tau_{i}\), which is proved to improve the cooperation between groups [27].  
Due to the different agent numbers in heterogeneous groups, for convenience of implementation, we calculate the mean value of \(\theta_{i}^{pred}\) and \(\theta_{i}\) for each group. For an agent group \(k\), \(\theta_{k}^{pred}\) is the mean value across all agents in \(k\), while \(\theta_{k}\) is the mean value of all agents that are **not** in group \(k\). Finally, since the GAS guarantees the capability of achieving the global optimality under the parameter-sharing constraint, the CGP loss is calculated by the _mean squared error_ (MSE) loss between \(\theta_{k}^{pred}\) and \(\theta_{k}\) for all agents and groups with the network parameter \(\mathbf{\psi}=(\psi_{1},\dots,\psi_{n})\), and is written as:  
\[\mathcal{L}_{CGP}(\theta_{\mathbf{\psi}}^{pred}|h)=\mathbb{E}_{\mathcal{D},K}( \theta_{k}^{pred}-\theta_{k})^{2}\;, \tag{6}\]  
where \(\mathbb{E}_{\mathcal{D},K}\) means to sample a batch of tuples \((\mathbf{\tau},s,r)\) from replay buffer \(\mathcal{D}\) and calculate expectation across the batch, and then calculate the mean values for all groups \(K\). For notational convenience, we also use \(\mathbb{E}_{\mathcal{D},K}\) for PPO-style on-policy methods without distinction. In that case, the replay buffer \(\mathcal{D}\) only consists of the present episode.  
Similarly, for value-based algorithms, since the action policy is usually calculated by the _argmax_ of Q values, the CGP loss is calculated by the MSE loss between \(Q_{k}^{pred}\) and \(Q_{k}\) for all agents and groups, and is written as:  
\[\mathcal{L}_{CGP}(Q_{\mathbf{\psi}}^{pred}|h)=\mathbb{E}_{\mathcal{D},K}(Q_{k}^{ pred}-Q_{k})^{2}\;. \tag{7}\]