# Target-Oriented Multi-Agent Coordination with Hierarchical Reinforcement Learning
## 4 Method
### Mechanism Description  
The regulatory hierarchical multi-agent coordination (RHMC) architecture is a two-level framework designed to enhance multi-agent coordination through hierarchical policies. At the high level, the policy collects joint state information from all agents and distributes target assignments to the low-level policy. The low-level policy then generates primitive actions based on both the assigned targets and the agents' observations, directly applying these actions to the environment. This hierarchical approach allows for more efficient and organized coordination among agents. The high-level actors focus on strategic target distribution, while the low-level actors handle tactical execution. Additionally, the high-level critic utilizes a regulation mechanism (RM) to calculate the value, ensuring robust and stable learning across both policy levels. RHMC has two Markov decision processes (MDPs) for the high-level policy and the low-level policy, respectively:  
\[MDP_{h}=\langle S_{h},G,P_{h},r_{h},\gamma_{h}\rangle\]  
\[MDP_{i}=\langle S_{l},A_{l},P_{l},r_{l},\gamma_{l}\rangle\]  
1. High-Level Policy (Symbol 1): The high-level policy is responsible for target allocation. It considers the overall state of the environment and assigns targets to agents. This level focuses on global coordination and strategic planning, ensuring that each agent is directed towards a target that optimizes the overall system performance. 2. Low-Level Policy (Symbol 2): The low-level policy deals with the execution of actions. Based on the assigned targets and local observations, agents choose primitive actions. This level emphasizes local decision-making and immediate interactions with the environment. The RHMC algorithm increases sample utilization efficiency via hierarchical policy training. However, the high-level policies in the hierarchical structure can be influenced by the low-level policies. When policies of all levels are trained simultaneously, the low-level policies are continually updated, leading to changes in the high-level transfer function. In such a non-stationary environment, it becomes challenging for high-level policy to learn the optimal policy. While the low-level policies \(\mu_{\theta}^{I}(s,g)\) interact directly with the environment, the high-level policy interacts indirectly. The non-stationarity emerges in the high-level policy \(\mu_{\theta}^{h}(s)\)  
Figure 1: Regulatory hierarchical multi-agent coordination (RHMC) architecture. transition. The low-level policy makes primitive actions directly on the environment, and the same state transition function remains unchanged. In contrast, the high-level policy acts indirectly on the environment. The same action output by the high-level policy can produce different actions at the low-level during training, leading to dynamic changes in state transitions at the high-level. This uncertainty in transitions \((s,g)\to s^{\prime}\) leads large changes in the high-level reward \(r_{h}\). To address these challenges, we propose a reward regulation mechanism and a target-conditioned filter:  
1. Both Regulation Mechanism: This mechanism aims to stabilize the training process by reducing the high-level policy's dependence on modifications in low-level behavior. By implementing a regularization term in the reward function, we ensure that the high-level policy remains robust despite the dynamic changes at the low-level. This helps maintain consistent learning and prevents the high-level policy from being adversely affected by the non-stationarity of the low-level policies. 2. Target-Conditioned Filter: The target-conditioned filter helps in refining the action selection process by incorporating target-specific information into the decision-making framework. This filter ensures that the actions chosen by the low-level policy are more aligned with the high-level targets, thus improving coordination and efficiency.