## 4 Method
### Practical Algorithms: G-MAPPO and G-QMIX
#### 4.3.2 G-QMIX  
Similar to the G-MAPPO, we combine the GAS with the QMIX [31] to form the G-QMIX algorithm. The network structure is referred to in Figure 3, and the pseudo-code of G-QMIX is given in Algorithm 2. In general, there are two parts of sub-networks optimized with different losses. The Q network includes the local Q network \(\mathbf{\theta}_{i}(Q|\tau)=(\theta_{1}(Q_{1}|\tau_{1}),\dots,\theta_{n}(Q_{n}| \tau_{n}))\) and the mixing network \(\mathbf{\theta}_{M}(Q_{i},s)\). The predictor network is symbolized with \(\mathbf{\psi}(Q^{pred}|h)=(\psi_{1}(Q_{1}^{pred}|h_{1}),\dots,\psi_{n}(Q_{n}^{pred} |h_{n}))\). Figure 3: An overall framework of G-QMIX. The overall network consists of two parts: Q network \(\mathbf{\theta}\) and predictor network \(\mathbf{\psi}\). The Q network includes the local Q network \(\mathbf{\theta}_{i}\) and the mixing network \(\mathbf{\theta}_{M}\). The local Q network \(\mathbf{\theta}_{i}\) and the predictor network \(\mathbf{\psi}\) are shared by all agents. They take their corresponding inputs and generate the GAS Q values \(\mathbf{Q}_{uni}\). Then, the \(\mathbf{Q}_{uni}\) are masked by different available action masks \(AM\) to generate different Q values for calculating different losses. The mixing network \(\mathbf{\theta}_{M}\) is a global network to compute the \(Q_{tot}\) during training. The local Q network \(\mathbf{\theta}_{i}\) is shared by all agents. It uses the current observation \(o_{i}^{t}\), the last action \(a_{i}^{t-1}\) and the last GRU hidden state \(h_{i}^{t-1}\) to calculate the next Q value \(Q_{\mathbf{\theta}_{i}}^{t}\). Similarly, we rotate the observation-action trajectory as \(\tau_{i}^{t}=(o_{i}^{t},a_{i}^{t-1},h_{i}^{t-1})\). Mixing network \(\mathbf{\theta}_{M}\) is a global network for value factorization. It takes all local \([Q_{\mathbf{\theta}_{i}}]\) as input, and mixes with the state \(s\) to produce the \(Q_{tot}\). The loss of \(\mathbf{\theta}\) is calculated following (11), where the \(Q_{tot}^{tgt}\) is the target Q value calculated by the target Q network \(\mathbf{\theta}^{tgt}\). Predictor network \(\mathbf{\psi}\) is used to compute \(Q_{\mathbf{\psi}}^{pred}\) and is optimized by the CGP loss function (7). The total loss function of G-QMIX is written as (12). Similar to (10), the \(\lambda_{I}\) is the coefficient hyper-parameter responsible for the CGP loss. \[ \begin{split}&\mathcal{L}(Q_{\mathbf{\theta}})\ =\ \mathbb{E}_{\mathcal{D}}[(y^{\mathbf{\theta}}-Q_{tot}(\mathbf{\tau},s;\mathbf{\theta}))^{2}] \,\\ & y^{\mathbf{\theta}}\ =\ r+\gamma\ \max_{\mathbf{a}^{t}}Q_{tot}^{tgt}(\mathbf{ \tau}^{\prime},s^{\prime};\mathbf{\theta}^{tgt})\,\\ &\mathcal{L}_{G-QMIX}=\mathcal{L}(Q_{\mathbf{\theta}})+\lambda_{I} \mathcal{L}(Q_{\mathbf{\psi}}^{pred})\.\end{split}  \tag{11}\]  
**Input**: Learning rate \(\alpha\), loss weight \(\lambda_{I}\), number of groups \(K\), number of agents \(N\), max total time-steps \(T^{TOT}\), max episodic time-steps \(T^{EP}\), batch size \(B\). **Initialize**: Network parameters \(\mathbf{\theta}\) and \(\mathbf{\psi}\), replay buffer \(\mathcal{D}=\{\}\), total running time-step \(t^{TOT}=0\). ```
1:while\(t^{TOT}\leq T^{TOT}\)do
2: Set episode step \(t=0\), receive initial state and observation \((s^{0},\mathbf{o}^{0})\) from the environment. 3:while\(t\leq T^{EP}\) and not terminateddo
4: Choose joint action \(\mathbf{a}^{t}\) using agent network \(\mathbf{\theta}_{i}\). 5: Interact with the environment using \(\mathbf{a}^{t}\) and receive reward \(t^{t}\), is terminated and next state and observation \((s^{t+1},\mathbf{o}^{t+1})\). 6: Collect a transition tuple at \(t\) and update the replay buffer \(\mathcal{D}=\mathcal{D}\cup\{(\mathbf{\tau}^{t},s^{t},t^{t})\}\). 7:\(t=t+1\). 8:endwhile
9: Sample a random batch of \(B\) episodes from \(\mathcal{D}\). 10: Calculate \(Q_{tot}\) using mixing network \(\mathbf{\theta}_{M}\). 11: Calculate joint predictive Q value \(Q_{\mathbf{\psi}}^{pred}\). 12: Calculate \(\mathcal{L}(Q_{\mathbf{\theta}})\) following (11).