# Target-Oriented Multi-Agent Coordination with Hierarchical Reinforcement Learning
## 5 Experiments
### Results and Comparisons  
In this experiment, different reinforcement learning algorithms were employed to tackle the cooperative navigation task, and their effectiveness was evaluated by comparing their reward averages. The results of the experiment, as shown in Figure 3, demonstrate that the RHMC algorithm performed the best and obtained the highest reward average for this task. This is because the RHMC algorithm emphasizes the coordination between high-level strategies, effectively coordinating agents and goals, and thus improving the global reward. By contrast, optimization methods based on global rewards, such as the QMIX and COMA algorithms, did not perform as expected. Although these methods can handle collaboration among different agents, they face the problem of conflict between local and global rewards. Therefore, these methods even performed worse than the IQL algorithm, which only ignores other agents. This suggests that global reward optimization methods like QMIX and COMA require better solutions for the conflict between local and global rewards. Overall, this experiment demonstrates the importance of coordination between high-level strategies for the cooperative navigation task. The RHMC algorithm performs well in coordinating agents and goals to improve global rewards. However, for global reward optimization methods like QMIX and COMA, it is necessary to resolve the conflict between local and global rewards to enhance their performance. The RHMC algorithm is not limited to cooperative navigation experiments with static targets but also performs well in multi-sensor multi-target coverage experiments with dynamic targets. As shown in Figure 4, after 5 million time steps of training in an environment with four agents and five targets, the RHMC algorithm obtained the highest global reward. This indicates that the high-level coordination of the RHMC algorithm is also applicable in multi-sensor multi-target coverage experiments and can improve global rewards. Figure 4: The model of regulation algorithm. Figure 3: The frame of low-level. Some methods based on local rewards, such as the MADDPG and MAPPO algorithms, did not perform well in this task. Although they used global information to maximize the local rewards of each agent, they still could not reach the performance level of the RHMC algorithm. In contrast, the HIT-MAC and QMIX algorithms ultimately converged to around 60 rewards, while the MADDPG and MAPPO algorithms reached the highest point of their learning curve after 1.5 million time steps. These results suggest that global reward optimization methods are more suitable than local reward optimization methods for solving the task in multi-sensor multi-target coverage experiments. During the converging phase, the RHMC algorithm achieved better global rewards than other algorithms through the coordination of high-level strategies. This is because the agents performed well in initial target allocation, but in the later stages of agent training, the strategies were bottlenecked by internal target conflicts. Therefore, the coordination of the RHMC algorithm largely improved the performance of the agents and achieved higher global rewards. This indicates that the coordination of high-level strategies is important for optimizing global rewards in multi-agent systems. As shown in Figure 5, we investigated the scalability and adaptability of the RHMC algorithm in the multi-target multi-sensor target coverage task with respect to the number of agents. We conducted experiments with different numbers of agents, compared the RHMC algorithm with other baseline algorithms, and observed the performance of the RHMC algorithm in different scenarios. The experimental results in Figure 6 demonstrate that the RHMC algorithm exhibits good scalability and adaptability in the presence of different numbers of agents. Regardless of an increase or decrease in the number of agents, the RHMC algorithm can adapt to different situations and achieve the highest global rewards.