# Multiagent Hierarchical Cognition Difference Policy for Multiagent Cooperation
## 5 Simulation Results and Analysis
### Cooperative Navigation  
For the cooperative navigation, there are \(N\) agents and \(N\) landmarks in the environment as shown in Figure 6a. The objective is for the agents to deploy themselves in a manner such that every agent reaches a distinct landmark. Note that we do not assign a particular landmark to each agent, but instead let the agents communicate with each other and develop a consensus as to who goes where. The compound reward function for this task is defined as follows:  
\[ \begin{array}{lcl}R^{i}_{total}&=&R^{i}_{dist}+R^{i}_{collision}+R_{cross}\\ R^{i}_{dist}&=&-dist\Big{(}p^{t}_{i},p^{g}_{i}\Big{)}\\ R^{i}_{collision}&=&\left\{\begin{array}{ll}0&if\ dist(p^{t}_{i},p^{t}_{j})>r_{ i}+r_{j}\\ -1&else\end{array}\right.\end{array}  \tag{26}\]  
where \(R_{dist}\) is the distance reward and \(R_{collision}\) is the collision reward. \(r_{i}\) represents the radius of agent \(i\) and \(p^{g}_{i}\) represents the goal position of agent \(i\). The closer the agent is to the goal point, the greater \(R_{dist}\) is. In this paper, there are four different scenarios with different numbers of agents. Specifically, these scenarios include scenario (a) with 6 agents, scenario (b) with 15 agents, scenario (c) with 20 agents, and scenario (d) with 29 agents. MA-HCDP is compared with MADDPG and TRANSFER in these scenarios. \begin{table}
\begin{tabular}{c c} \hline \hline
**Parameters** & **Value** \\ \hline Learning rate & 0.0001 \\ \hline Max gradient normalization & 2 \\ \hline Discount factor \(\gamma\) & 0.99 \\ \hline Coefficient of value loss function \(\beta_{1}\) & 0.5 \\ \hline Coefficient of policy loss function \(\beta_{1}\) & 1 \\ \hline Coefficient of entropy \(\beta_{3}\) & 0.01 \\ \hline Coefficient of VAE loss function \(\beta_{4}\) & 0.01 \\ \hline Episode & 20,000 \\ \hline Batch size & 64 \\ \hline PPO epoch & 4 \\ \hline Number of attention heads in HGN & 3 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Parameters of training. The learning curves of all the approaches in terms of mean rewards are presented in Figure 7. The coordinate name of the \(x\)-axis indicates the time spent in training. The longer the time, the slower the method will converge. The coordinate name of the \(x\)-axis represents the reward obtained in an episode. The higher the reward, the better the performance of the method. As shown in Figure 7, MA-HCDP has a higher convergence rate and higher performance than MADDPG and TRANSFER. The results indicate that different influences of observations of different groups of agents can be handled by MA-HCDP. It demonstrates that the redundant communication information can be filtered effectively by MA-HCDP, which enable agents to learn more appropriate policy in fewer training episodes. Note that MADDPG converges faster than TRANSFER but converges to a minimum value. As a comparison, TRANSFER converges slower but obtain higher rewards than MADDPG. Although TRANSFER considers the influence of communication among different agents, it ignores the influence of redundant communication, which make agents trained with TRANSFER obtain higher rewards and converge slower than MADDPG. Due to the ignorance of the influence of redundant communication, although the agent trained by TRANSFER can learn appropriate policy, its performance and convergence speed are both weaker than MA-HCDP. The above analysis shows the effectiveness of our method in dealing with the influence of different groups of agents and redundant communication. In addition to the training process data, the simulation results of 100 independent simulations for each scenario are presented in Table 2. The \(t\)-test value is adopted to evaluate the effectiveness of our method statistically. According to the mean value, standard deviation and the sample data for 100 tests, we calculate the \(t\) value of MA-HCDP and the other methods on the same test scenario.