## 2 Materials and Methods
### Proposed Method
#### 2.3.4 Implementations  
QMIX-GNN employs an end-to-end training approach, with its overall training architecture aligning with DQN. The selected loss function is as follows:  
\[\mathcal{L}(\theta)=\sum_{i=1}^{b}\Bigl{[}\left(y_{i}^{tot}-Q_{tot}(\mathbf{o}, \mathbf{u},s;\theta)\right)^{2}\Bigr{]} \tag{15}\]  
where \(b\) denotes the batch size sampled from the replay buffer, \(\mathbf{o}\) represents the joint observation of agents, \(\mathbf{u}\) is the joint action of the agents, \(s\) represents the absolute global state, \(\theta\) represents the parameters of the estimated network, and \(y^{tot}\) is the output of the target network:  
\[y^{tot}=r+\gamma\max_{u^{\prime}}Q_{tot}\left(\mathbf{o}^{\prime},\mathbf{u} ^{\prime},s^{\prime};\theta^{-}\right) \tag{16}\]  
where \(\theta^{-}\) stands for the parameters of the target network. The pseudocode of our QMIX-GNN is provided in Algorithm 1. \begin{table}
\begin{tabular}{c c} \hline \hline
**Variable** & **Meaning** \\ \hline \(o\) & The agent’s raw observation \\ \(\phi\) & The agent’s type \\ \(\sigma^{\prime}\) & The projected feature \\ \(b\) & The aligned vector output of Shared-Bottom \\ \(\mathbf{M}\) & The projection matrix \\ \(o_{all}\) & The team information \\ \(h\) & Attention head \\ \(H\) & The number of attention heads \\ \(N\) & The number of agents \\ \(\mathcal{N}\) & The number of \(k\)-nearest neighbors \\ \(a\) & Agents’ actions \\ \(u\) & The joint action of agents \\ \(s\) & Agents’ states \\ \(r\) & Agents’ rewards \\ \(Q\) & Agents’ Q-value \\ \(\theta\) & Models’ parameters \\ \(v\) & The feature that integrates neighbor information \\ \(z\) & The concatenated features \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Important variables and their meanings. Figure 7: Structure of the mixing network. ```
1:Hyperparameters. 2:Model parameters \(\theta\). 3:\(step=0\), \(\theta^{-}=\theta\). \(\triangleright\) Initialize
4:while\(step<step_{max}\)do
5:\(t=0\)\(\triangleright\) Initialize episode
6:\(s_{0}\gets initial\)\(\triangleright\) Obtain the initial state
7:while\(s_{i}\
eq\) terminal and \(t<\) episode limit do
8:\(\mathbf{o}_{t}\leftarrow\) observations for all agents from the environment
9:\(A_{t}\gets\)\(K\)-Nearest Neighbor Graph \(\triangleright\) Construct the graph for current timestep
10:\(o_{all}^{t}\leftarrow\)\(\operatorname{GAT}(\mathbf{o}_{t},A_{t})\)\(\triangleright\) Infuse observation information from all agents
11:for each agent \(i\)do
12:\(\epsilon=\) epsilon-schedule\((step)\)\(\triangleright\) Explore or exploit threshold
13:if\(\operatorname{random}(0,1)<\epsilon\)then\(\triangleright\) Explore or exploit
14:\(Q_{i}\leftarrow\operatorname{RNN}\bigl{(}o_{i}^{t},a_{i}^{t},a_{all}\bigr{)}\)
15:\(a_{i}^{t}=\operatorname{arg\,max}_{a^{t}}Q_{i}(a^{t})\)
1



meaning that the environment provides global observations during training, but such information is not available to agents during execution. The SMAC offers several scenarios, including homogeneous and heterogeneous ones. Homogeneous scenarios involve both sides having only one type of agent. For example, in the 3m (Marine) scenario, both allies and enemies have only Marines as agents. On the other hand, heterogeneous scenarios feature multiple types of agents with different skills and attributes. For instance, in the 2s3z (Stalker+Zealots) scenario, both sides consist of two Stalkers and three Zealots. In this experiment, we used a computer with an Intel(R) Xeon(R) Gold 6330 CPU @ 2.00 GHz processor and an RTX 3090 GPU, running on the Ubuntu operating system. The experiments were conducted using the Python programming language and the PyTorch deep learning framework. The specific versions of the hardware and software used in the experiment are shown in Table 2.

### Experimental Configure

To comprehensively validate the performance of QMIX-GNN, experiments were conducted on six scenarios in the SMAC platform: 3m, 2s3z, 1c3s5z, 10m_vs_11m, MMM, and MMM2, which include both homogeneous and heterogeneous scenarios.