## 5 Experiments and Results
### Experimental Details
#### 5.1.2 Validating Tests in the MPE, MAMuJoCo, and RPE Environments  
In the MPE validating tests, the goal of scenario large_spread is to control agents of different groups to approach their different goals. The goals are the small circles indicated by red arrows in Figure 4B, and the agents from the same group are marked with the same colors. We conduct experiments in two scenarios, large_spread_v1 and large_spread_v2. There are five agents in each group for all three groups in large_spread_v1, and there are ten agents in each group for all three groups in large_spread_v2. In general, the agent actions are five discrete numbers indicating _stop, moving up, down, left and right_, and the agents and goals are stochastically initialized at different positions at the start of every episode. More detailed information can be found in [35; 38]. We turn off the collision punishment and all agents share the same global reward, which is calculated with the sum of distances of all agents towards their respective goals. The overall training steps are 2M for value-based methods and 5M for policy-based methods. The \"steps-of-an-episode\" is 50 for all tests. We use the _return_ as the measuring criterion. The average value of _return_ over 32 testing episodes for every 10,000 training steps is used for plotting. We take four rounds of individual tests with different random seeds. MAMuJoCo extends the well-known MuJoCo physics-based simulation environment to support multiple agents for MARL studies. We choose G-MAPPO and MAPPO to run experiments in \(HalfCheetah-2\times 3\) and \(Walker-6\times 1\) scenarios. In the \(HalfCheetah-2\times 3\) scenario, there are a total of six agents. The six agents are divided into two groups with three agents each. Similarly, in the \(Walker-6\times 1\) scenario, there are six agents in total, divided into six groups with one agent each. Each agent has its own observation space. The observations typically include information about the agent's own state, such as joint angles, joint velocities, and body positions. The action space for each agent consists of a set \begin{table}
\begin{tabular}{c c c c c} \hline
**Map Name** & \begin{tabular}{c} **Ally\_** \\ **Marines** \\ \end{tabular} & \
\begin{tabular}{c} **Ally\_** \\ **Medivacs** \\ \end{tabular} &
\begin{tabular}{c} **Enemy\_** \\ **Marines** \\ \end{tabular} & **Difficulty** \\ \hline
6m2m\_15m & 6 & 2 & 15 & Easy \\
6m2m\_16m & 6 & 2 & 16 & Medium \\
8m3m\_21m & 8 & 3 & 21 & Medium \\
8m4m\_23m & 8 & 4 & 23 & Hard \\
12m4m\_30m & 12 & 4 & 30 & Ex-Hard \\ \hline \end{tabular}  
* The name \(XmYm\_Zm\) means that the map consists of \(X\) ally marines, \(Y\) ally microvacs, and \(Z\) enemy marines.  
\end{table} 
Table 3: Asymmetric heterogeneous SMAC maps. of real-number motor commands that can be used to control the body. These commands can adjust the torque applied to each joint, enabling agents to move their legs, torso, and other body parts. Rewards are proportional to the forward distance of the body. More detailed information can be accessed in [40]. The overall training steps total 10M for the two methods. The \"steps-of-an-episode\" is 200 for all tests. We also use the average value of _return_ over 32 testing episodes for every 5000 training steps for measuring. Three rounds of individual tests are taken with different random seeds. The curves of _return_ changing over time are plotted with the \(p\)-value being 0.05 and the confidence intervals being 95%. The Resource Preemption Environment (RPE) [41] is designed to simulate real-world manufacturing scenarios where resources are limited and job scheduling is complex. It is a crucial part of the research on solving job scheduling problems in a resource-constrained environment. In the RPE, multiple robots share a common pool of resources for operation processing. For example, in the automobile vice-frame processing case, there are eighteen robot sites, and some of them share the same pool of resources. Each resource can only be used by one robot at a time.