## 2 Related Works  
### Multi-Agent Reinforcement Learning and Heterogeneity  
The centralized training with decentralized execution (CTDE) paradigm [42; 43; 44], which requests agents not to use state \(S\) during execution, has become one of the most successful MARL paradigms in the SMAC environment.  
For policy-based methods, a simple implementation of CTDE is the \"centralized critic and decentralized actor\" (CCDA) structure. The state \(S\) is only available for the critic network during the centralized training phase. COMA [45] uses a counter-factual baseline for learning a better policy. MAPPO [32] directly implements the CCDA structure with a global critic and a shared actor. HAPPO [28] introduces the \"multi-agent advantage decomposition\" (MAAD) lemma and designs a sequential executing and updating structure for the independent actors. MAT [46] further develops the HAPPO by replacing the backbone of HAPPO with the Transformer-style network structure, and naturally uses the auto-regressive decoder as the sequential actor.  
For value-based methods, the mainstream is the value factorization method, whose goal is to learn a centralized and factorized joint action-value function \(Q_{tot}\) and the factorization structure \(Q_{tot}\to Q_{i}\). In order to factorize \(Q_{tot}\) and use the _argmax_ policy of \(Q_{i}\) to select actions, Individual-Global-Max (IGM) consistency [47] is required.  
QMIX [31] dramatically enriches the embedding ability of the network with the use of a monotonic constraint. RODE [48] and ROMA [49] are _role-based_ algorithms. In general, several key hyper-parameters define the clustering and usage of role policies. Therefore, the performance of ROMA and RODE highly depends on searching for the optimal hyper-parameters, not to mention that the default training step of ROMA is 20M time-steps.  
UNMAS [50] proposes the self-weighting mixing network and the individual action-value networks to adapt to the changes in agent numbers and action space.  
Traditionally, the study of heterogeneity in MARL is insufficient, because heterogeneous MARL has been considered as a special case of homogeneous MARL. HAPPO lacks specific analysis and sufficient experiments to analyze the effect of heterogeneity. Its successors, MAT and HASAC [51], only modify the network structure and optimizing methods and still lack the study of heterogeneity. SHPPO [52] uses a latent network to learn embedding variables for heterogeneous decision making. In the other field of MAS, FedMRL [53] develops a novel federated multi-agent deep reinforcement learning framework to address data heterogeneity for medical imaging. HeR-DRL [54] uses a customized heterogeneous Graph Neural Network (GNN) to capture the pair-wise interaction relationships within the robot-crowd and then optimizes the overall network with a DQN-style DRL method.  
Recently, HetGPPO [24] conducts a taxonomical study on heterogeneity classification, and introduces a graph neural network heterogeneous agent-wise communication. GHQ [27] gives a formal definition of the _local transition_ heterogeneity, which is equal to the _physical_ heterogeneity, and uses a grouping method and hybrid value factorization structure to learn grouped Q functions for choosing actions.