# Target-Oriented Multi-Agent Coordination with Hierarchical Reinforcement Learning
###### Abstract  
In target-oriented multi-agent tasks, agents collaboratively achieve goals defined by specific objects, or targets, in their environment. The key to success is the effective coordination between agents and these targets, especially in dynamic environments where targets may shift. Agents must adeptly adjust to these changes and re-evaluate their target interactions. Inefficient coordination can lead to resource waste, extended task times, and lower overall performance. Addressing this challenge, we introduce the regulatory hierarchical multi-agent coordination (RHMC), a hierarchical reinforcement learning approach. RHMC divides the coordination task into two levels: a high-level policy, assigning targets based on environmental state, and a low-level policy, executing basic actions guided by individual target assignments and observations. Stabilizing RHMC's high-level policy is crucial for effective learning. This stability is achieved by reward regularization, reducing reliance on the dynamic low-level policy. Such regularization ensures the high-level policy remains focused on broad coordination, not overly dependent on specific agent actions. By minimizing low-level policy dependence, RHMC adapts more seamlessly to environmental changes, boosting learning efficiency. Testing demonstrates RHMC's superiority over existing methods in global reward and learning efficiency, highlighting its effectiveness in multi-agent coordination. multi-agent; target-oriented; hierarchical reinforcement learning; coordination 2024147084.103390/app14167084.10390/app14167084. case, coordination involves managing and optimizing server resources to handle various tasks or requests in a balanced and efficient manner [17; 18]. Coordinating the relationships between agents and targets is crucial for improving the overall efficiency of collaborative tasks [19; 20]. For one-to-one agent-target tasks, the IQL method treats the remaining agents directly as part of the environment, leading to instability as the policies of other agents change over time [17; 21]. Since there is no guarantee of convergence, it is easy for the agent to become stuck in an endless exploration, limiting IQL to simple tasks such as the Pong game [22]. However, in cooperative tasks with multiple targets, such as cooperative navigation, where agents must reach multiple target locations, it is critical to coordinate the relationship between agents and targets to avoid internal conflicts [23; 24]. MADDPG learns multiple policies for each agent and optimizes the overall effect of all policies to improve the stability and robustness of the algorithm [18]. COMA utilizes a global critic network to assess all current actions and states, enhancing the efficiency of information exchange and collaboration between agents during training [19]. QMIX employs a hybrid network to combine the single agent's local value function with global state information, facilitating training and learning procedures, and enhancing the algorithm's performance [20]. In multi-agent coordination, maximizing local rewards does not ensure global optimality. In contrast, prioritizing the global reward can cause entanglement between local and global rewards, leading to suboptimal solutions [21]. Balancing the relationship between the local and the whole is very important for multi-agent systems to accomplish the task better [22; 24]. The regulatory hierarchical multi-agent coordination (RHMC) framework is proposed to tackle this problem. Our contribution is threefold: (1) A hierarchical reinforcement learning algorithm is proposed for the multi-agent coordination problem. This framework comprises two layers; the high-level policy governs target allocation, while the low-level policy guides agent actions. (2) RHMC implements a reward regularization mechanism to reduce the high-level policy's dependence on modifications in low-level behavior, providing stability to the policy training process.