# Target-Oriented Multi-Agent Coordination with Hierarchical Reinforcement Learning
## 2 Related Work  
The target-oriented multi-agent coordination framework includes two categories of multi-agent reinforcement learning: local reward-based coordination and global reward-based coordination. Local reward-based approaches aim to optimize overall performance by maximizing the individual agent's local rewards. Early approaches, such as the IQL method [17], convert multi-agent-related tasks into single-agent tasks, with each agent learning its own value function separately. This method is not effective because all agents affect the overall environment, and the agents are also affected by other agents during the learning process, leading to unstable agent learning. The Centralized Training with Decentralized Execution (CTDE) framework is employed in multi-agent learning work such as intelligent transportation [21], smart home [22], recommendation algorithms [23; 24; 25], common-pool resource appropriation [26; 27; 28], and sequential social dilemmas [29; 30]. This approach addresses the issue of non-stationarity in the environment that can arise when agents learn independently without taking into account the actions of other agents. The MADDPG method [18] can be regarded as a deterministic policy gradient method that uses deep reinforcement learning to solve the continuous action space problem. Although MADDPG enhances stability and robustness, it still faces challenges in diverse environments. The M3 DDPG method [31] enhances the generalization capability of the learned policies and accommodates the heterogeneity of reward functions across agents, but it does not explicitly address the coordination requirements in multi-agent systems. The MAPPO method [32] is a multi-agent reinforcement learning method based on the policy gradient algorithm of confidence region, which limits the size of the parameter update step to stabilize the training process. The NC-HDQN method [33] analyzes the correlation between agents and their connected neighbors to measure the observed value and reward and uses a hysteretic DQN-weighted information optimization strategy. The LOLA method [34] enables agents to better understand the policies and motivations of other agents through opponent modeling, thus coordinating their actions more effectively. Global reward-based approaches tend to have agents with one goal and a shared global reward. These approaches aim at maximizing the global reward while optimizing the local reward. The COMA method [19] uses a centralized critic network to collect information from all agents during training and employs counterfactual baselines to address the credit assignment problem. Value decomposition methods represent the joint action value function as a combination of the value functions of each agent, guiding agents to learn a better policy. The VDN method [35] uses a deep neural network to represent the value function of each agent and the whole joint action value function as a linear combination of these value functions. However, VDN overlooks the role of global state information in the learning process and the intricate nonlinear relationships between agents. To address these issues, the QMIX method [20] uses an additional network to fuse the values of each agent, including global information as auxiliary information when expressing the joint action value function. QMIX obligates the joint action value function to be monotonic with the value function of each agent to comply with the maximum consistency principle between corresponding value functions. The PMI-Q method [36] improves the generalization ability of the policy by preventing overfitting through a combination of pessimistic averaging and average invariance. Drawing inspiration from the collective behavior of ant colonies, the SIRL method [37] introduces the concept of pheromones into multi-agent systems to reduce local conflicts. Hierarchical reinforcement learning (HRL) benefits structured exploration by learning sub-task policies over primitive actions. Exploration efficiency is essential in reinforcement learning.