## 2 Materials and Methods
### Proposed Method
#### 2.3.1 Agent Network  
The agent network is designed to compute each agent's local Q-value. It consists of three parts in total, as shown in Figure 3:  
* A projection matrix is used to map the states of heterogeneous agents into a consistent state space.
* Shared-bottom parameters accelerate the model's learning speed and improve its generalization ability.
* A tower structure enables different types of agents to learn differentiated policies, helping them to adapt to various action spaces.  
Figure 2: Structure of QMIX-GNN.  
In MARL, the feature spaces and actions of heterogeneous agents may be inconsistent, necessitating the processing of this heterogeneous information. For cases where the feature spaces are inconsistent, a transformation matrix is constructed for each type of agent to map the agents' heterogeneous features into a common feature space. The projection process is shown in Equation (4):  
\[o_{i}^{\prime}=\mathbf{M}_{\phi_{i}}\cdot o_{i}, \tag{4}\]  
where \(\phi_{i}\) represents the type of agent \(i\), \(\mathbf{M}\) is the projection matrix, \(o_{i}\) is the agent's raw observation, and \(o_{i}^{\prime}\) is the projected feature of the agent derived from its raw observation. With a projection matrix specific to each type, the features of any agent can be processed.  
To address the issue of inconsistent action spaces among heterogeneous agents in MARL, we adopt the tower structure from multitask learning. This structure employs a shared bottom in order to share the lower-level parameters among multiple task models. At the same time, it uses independent towers to produce different outputs for the specific differences of each task [3]. This approach can significantly reduce the number of parameters and the computational complexity, improving the training speed and generalization ability of the model. To account for the differences among individual agents, independent towers can be utilized to generate distinct outputs to accommodate various action spaces, effectively modeling the agents' heterogeneity and enhancing the decision-making performance of the multi-agent system.  
The shared bottom layer takes the aligned information projected from the agent along with team information \(o_{all}\) and outputs the aligned vector \(b_{i}\), as follows:  
\[b_{i}=\text{bott}\big{(}\text{concat}\big{(}[o_{i}^{\prime},o_{all}]\big{)} \big{)}, \tag{5}\]  
where bott represents the shared bottom layer common to all agents and concat denotes the concatenation operation. The method for obtaining \(o_{all}\) will be detailed later. In this paper, we employ an RNN to implement the shared bottom network with a tower structure designed for each agent to address the heterogeneous policy differences among agents and output distinct action spaces. Consequently, the individual Q-value of an agent can be expressed as  
\[Q_{i}=\text{tow}_{\phi_{i}}(b_{i}), \tag{6}\]  
where tow refers to the tower structure designed for agent \(i\) (implemented in this work as a Multi-Layer Perceptron (MLP) network) and \(Q_{i}\) denotes the Q-value of agent \(i\). In conclusion, the overall process of handling the heterogeneous feature and action spaces across agents can be formulated as  
\[Q_{i}=\text{tow}_{\phi_{i}}\big{(}\text{bott}\big{(}\text{concat}\big{(}[ \mathbf{M}_{\phi_{i}}\cdot o_{i},o_{all}]\big{)}\big{)}\big{)}. \tag{7}\]  
Figure 3: Structure of the agent network.