# Target-Oriented Multi-Agent Coordination with Hierarchical Reinforcement Learning
## 6 Conclusions  
In this paper, we propose the regulatory hierarchical multi-agent coordination (RHMC) framework to address coordination challenges in target-oriented multi-agent tasks. The RHMC framework introduces a two-level hierarchical reinforcement learning approach, where the high-level policy is responsible for target assignment and the low-level policy focuses on executing primitive actions. Our extensive experimental results in both static and dynamic target-oriented environments demonstrate the significant advantages of the RHMC framework over existing methods. Specifically, in the cooperative navigation task, the RHMC algorithm achieved an average reward of 70.5, significantly outperforming QMIX and COMA, which achieved average rewards of 55.1 and 35.6, respectively. As illustrated in Figure 6, RHMC maintained a higher reward trajectory throughout the training process, demonstrating its superior high-level strategy coordination and target assignment capabilities. In the dynamic multi-sensor multi-target coverage task, RHMC achieved a peak global reward of 92.1 after 5 million time steps, compared to 78.6 for QMIX and 75.2 for COMA, highlighting its adaptability to changing target dynamics. Figure 7 shows that RHMC's reward increased steadily, reaching higher rewards faster than other methods, thus proving its efficiency and robustness in real-time adaptation. Ablation studies further validate the importance of the regulation mechanism and target-conditioned filter in enhancing the RHMC's performance. The full RHMC method, incorporating both the regulation mechanism and the target-conditioned filter, achieved a final reward of 70.5, while the variants without these components achieved 55.1 and 35.6, respectively. This indicates that the regulation mechanism and target-conditioned filter contribute to a 50.9% and 21.4% performance improvement, respectively, as shown in Figure 10.  
The regulation mechanism effectively stabilized the high-level policy learning process, mitigating the non-stationarity introduced by dynamic changes in low-level policies. This stability, coupled with the target-conditioned filter's ability to reduce noise from irrelevant information, significantly enhanced coordination efficiency and overall system performance. In conclusion, the RHMC framework represents a substantial advancement in multi-agent reinforcement learning, particularly in target-oriented environments. By addressing \begin{table}
\begin{tabular}{c c c} \hline \hline
**Method** & **Final Reward** & **Performance Improvement** \\ \hline Full RHMC & 70.5 & N/A \\ Without regulation mechanism & 35.6 & 50.9\% \\ Without target-conditioned filter & 55.1 & 21.4\% \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Ablation study results. the coordination problem through a structured hierarchical approach, RHMC not only improves learning efficiency and robustness but also achieves superior performance metrics in both static and dynamic tasks. Future research could further explore the application of RHMC in more complex multi-agent systems, potentially extending its scalability and effectiveness. By integrating these innovations, RHMC sets a new benchmark for multi-agent coordination tasks, achieving unprecedented results in terms of efficiency and effectiveness compared to state-of-the-art methods. Conceptualization, Y.Y. and W.L. ; Methodology, Z.Z. ; Validation, Y.Y. and J.M. ; Formal analysis, W.L. and J.M. ; Investigation, Y.Y. ; Resources, W.L. and J.M. ; Data curation, W.L. ; Writing--original draft, Y.Y. ; Writing--review & editing, Z.Z. ; Supervision, Y.Y. and Z.Z. ; Project administration, Z.Z. All authors have read and agreed to the published version of the manuscript. This research was funded by the Guangxi Natural Science Foundation of China, grant number 2023 GXNSFAA026270, the Guangxi Science and Technology Project, grant number AD20159034, and the National Natural Science Foundation of China, grant numbers 62262009, 61902086, and 6202780103. The APC was funded by these grants. Not applicable.