# Target-Oriented Multi-Agent Coordination with Hierarchical Reinforcement Learning
## 5 Experiments
### Environmental Settings  
Cooperative Navigation. In this experiment, a group of n agents are required to reach n landmarks (5). The agents receive rewards based on the negative distance to their nearest landmark. In the event that an agent collides with another agent, it receives a penalty of \(-1\). Therefore, the agents must learn to cover all the landmarks while avoiding collisions. To sum up, when the agent k moves from state \(S_{i}\) to state \(S_{i+1}\), the reward that the environment gives back to the agent is as follows:  
\[r_{k}=-d_{k,s_{i+1}}-\sum_{j=1}^{n}P(k,j)\Bigg{|}_{k\
eq j} \tag{18}\]where \(P(i,j)\) represents whether the two agents collided \((0:No,1:Yes)\), and \(d_{k_{s|i+1}}\) represents distance to their nearest landmark. Finally, the reward of each agent will be summarized as the cooperation reward of the whole agent system. Each agent will use the global reward for training.  
Multi-sensor Multi-target Coverage Task. In this experiment, a group of n agents (sensors) need to adjust their angles to cover as many as possible of the m targets present in the environment. The sensors are fixed in place and must take an action at each time step to adjust their angle. The targets moved randomly in the environment, making it difficult for the sensors to effectively track them. In every time step, the agent obtains the observed state \(s_{i}=(s_{i,1},s_{i,2},\ldots,s_{i,m}).s_{i,j}\) describes the relative position information of the agent \(i\) and the target \(j.s_{i,j}=(i,j,a_{i,j},d_{i,j})\), where \(i\) and \(j\) represent the identification number of the agent and the target, respectively. \(\alpha_{i,j}\) and \(d_{i,j}\) represent the relative angle and absolute distance of the agent \(i\) and the target \(j\), respectively. Each agent took a primitive action at every time step, selecting from a discrete action space comprising turn right, turn left, and stay. The agent's rotation angle was fixed at 5 degrees. In RHMC, the action of the high-level action is the assigned target \(g\), which is a vector of length \(n.g_{i,j}=1\), which means target \(j\) is assigned to agent \(i\) and \(g_{i,j}=0\) means it is not.  
As agents' policies exhibit similar functionality, sharing parameters can reduce the computational complexity of learning, thereby improving learning efficiency and performance. To encourage varied behavior among the agents, the policy incorporates each agent's identity as a one-hot vector input. If the input sizes of agents vary, padding of zeros is incorporated to maintain consistent input dimensionality. If agents have varying numbers of operations, the probability of selecting invalid operations is set to zero. For each agent, losses are determined, and the shared parameters are adjusted accordingly. Random seeds were used for the experiments, with each experiment comprising a maximum of 300 episodes. The learning rate and discount factor used for the experiments were 0.0005 and 0.99, respectively.  
The following baselines were used to compare our results: HiT-MAC [37] is a hierarchical coordination architecture that uses pre-trained, maladaptive low-level policies. QMIX [20] is a multi-agent algorithm based on value decomposition. It uses two neural networks to deduce the global value by combining the Q value of each agent and the global state. IQL [17] regarded other agents as environments without considering their changes, while COMA [8] adopted a counterfactual baseline to solve the problem of credit allocation. MAPPO [32] uses an adaptive KL divergence to ensure stable policy updates during optimization. MADDPG [18] is a deep reinforcement learning-based multi-agent algorithm that uses experience replay and target networks to reduce the variance of the Q-value function.