## 4 Method
### Practical Algorithms: G-MAPPO and G-QMIX
#### 4.3.1 G-Mappo  
In order to implement the GAS based on policy-based methods, we combine the GAS with the MAPPO [32] to form the G-MAPPO algorithm. The detailed network structure is illustrated in Figure 2, and a pseudo-code is given in Algorithm 1. The overall network consists of three different sub-networks: the actor network \(\mathbf{\theta}(a|\tau)=(\theta_{1}(a_{1}|\tau_{1}),\ldots,\theta_{n}(a_{n}|\tau_{n}))\), the predictor network \(\mathbf{\psi}(\theta^{pred}|h)=(\psi_{1}(\theta_{1}^{pred}|h_{1}),\ldots,\)\(\psi_{n}(\theta_{n}^{pred}|h_{n}))\) and the critic network \(\mathbf{\phi}(V|s)\). These parameters are responsible for the joint action policy \(\pi_{\mathbf{\theta}}=(\pi_{\theta_{1}},\ldots,\pi_{\theta_{n}})\), the joint prediction policy \(\mathbf{\phi}_{\mathbf{\psi}}^{pred}=(\theta_{\psi_{1}}^{pred},\ldots,\theta_{\psi_{n }}^{pred})\) and the global value function \(V_{\mathbf{\phi}}(s)\), respectively. Following the implementation of MAPPO, the actor \(\pi_{\mathbf{\theta}}\) is shared by all agents and is optimized following the loss function (8). It receives the current observation \(o^{t}\), the last action \(a^{t-1}\), and the last GRU hidden state \(h^{t-1}\) to calculate the next action \(a^{t}\). Due to the analysis in Section 4.2, we notate the observation-action trajectory as \(\tau^{t}=(o^{t},a^{t-1},h^{t-1})\). The critic \(V_{\mathbf{\phi}}\) is a mapping function \(S\rightarrow\mathbb{R}\), and is centrally trained following the loss function (9). The additional predictor module \(\mathbf{\phi}_{\mathbf{\psi}}^{pred}\) is also shared by all agents for generating the prediction policy and is optimized following the CGP loss function (6). \[ \begin{split}\mathcal{L}(\pi_{\mathbf{\theta}})&=\mathbb{ E}_{\mathcal{D}}[\min(\frac{\pi_{\mathbf{\theta}}(a|\tau)}{\pi_{\mathbf{\theta}_{ old}}(a|\tau)}A_{\pi_{\mathbf{\theta}}}(s,a),\\ & clip(\frac{\pi_{\mathbf{\theta}}(a|\tau)}{\pi_{\mathbf{\theta}_{ old}}(a|\tau)},1\pm\epsilon_{p})A_{\pi_{\mathbf{\theta}}}(s,a))]\\ &+\lambda_{E}\mathbb{E}_{\mathcal{D}}[S(\pi_{\mathbf{\theta}}(\tau) )]\;.\end{split}  \tag{8}\]  
\[ \begin{split}\mathcal{L}(V_{\mathbf{\phi}})&=\mathbb{ E}_{\mathcal{D}}[\max((V_{\mathbf{\phi}}(s)-\hat{R})^{2},\\ &(clip(V_{\mathbf{\phi}}(s),V_{\mathbf{\phi}_{old}}(s)\pm\epsilon_{v}))- \hat{R})^{2})]\;.\end{split}  \tag{9}\]  
\[ \begin{split}\mathcal{L}_{G-MAPPO}&=\mathcal{L}(\pi _{\mathbf{\theta}})+\lambda_{V}\mathcal{L}(V_{\mathbf{\phi}})+\lambda_{I}\mathcal{L}( \theta_{\mathbf{\psi}}^{pred})\;.\end{split}  \tag{10}\]  
The overall loss function of G-MAPPO is (10). The _clip_ operator indicates clipping the value of the first variable to fit in the threshold interval determined by the second function with the hyper-parameter \(\epsilon_{p}\) or \(\epsilon_{v}\). \(A_{\pi_{\mathbf{\theta}}}(s,a)\) is the advantage value computed with the GAE [59] method. \(S(\pi_{\mathbf{\theta}}(\tau))\) is the policy entropy. \(\hat{R}\) is the discounted reward-to-go. \(\lambda_{E}\), \(\lambda_{V}\), and \(\lambda_{I}\) are the coefficients responsible for the entropy loss, value loss, and CGP loss, respectively. Figure 2: An overall framework of G-MAPPO. The overall network consists of three parts: actor network \(\mathbf{\theta}\), predictor network \(\mathbf{\psi}\) and critic network \(\mathbf{\phi}\). The actor network \(\mathbf{\theta}\) and the predictor network \(\mathbf{\psi}\) are shared by all agents. They take their corresponding inputs and generate the GAS policies \(\mathbf{\pi}_{uni}\). Then, the \(\mathbf{\pi}_{uni}\) are masked by different available action masks \(AM\) to generate the joint action policy \(\pi_{\mathbf{\theta}}\) and the joint prediction policy \(\theta_{\mathbf{\psi}}^{pred}\) separately. The critic network \(\mathbf{\phi}\) is a global network to compute the global value function \(V_{\mathbf{\phi}}(s)\) during training.