## 4 Simulation Experiment
### Experiment 1: Fixed Initial Position  
The fixed initial position experiment means that for every episode of training, it will start with the same positions of the agents, the obstacles, and the target. To show that the proposed MADDPG-E algorithm can help the multi-agent system achieve the objective in any situation, two fixed initial position experiments with different initial positions (as shown in Table 3) are performed. Figure 4a shows the multi-agent target search process under initial positions 1, and Figure 4b shows the results with initial positions 2. The figure shows that the agents find the target at 29 and 32 steps, respectively. It can be seen that the multi-agents can learn good cooperative search strategies through training. As it can be seen from the process of the agent searching for the target, that when facing a target moving faster, agents can use obstacles, boundaries, and other environmental factors to form an encircling strategy for the target so as to achieve the purpose of searching for the target. \begin{table}
\begin{tabular}{c c} \hline \hline
**Training Parameter** & **Values** \\ \hline Total number of episodes & 30,000 \\ Maximum episode length & 50 \\ Discount factor \(\gamma\) & 0.98 \\ Actor network learning rate \(\alpha_{a}\) & 0.01 \\ Critic network learning rate \(\alpha_{c}\) & 0.01 \\ Batch size & 1024 \\ Size of replay buffer \(M\) & \(10^{6}\) \\ \hline \hline \end{tabular}
\end{table} 
Table 2: The training parameters. \begin{table}
\begin{tabular}{c c} \hline \hline
**Elements** & **Initial Positions 1** & **Initial Positions 2** \\ \hline Agent1 & [\(-\)0.60, 0.30] & [0.45, \(-\)0.15] \\ Agent2 & [0.00, 0.50] & [0.60, 0.60] \\ Agent3 & [0.50, \(-\)0.20] & [0.65, \(-\)0.52] \\ Target & [0.00, 0.10] & [0.65, 0.00] \\ Obstacle1 & [\(-\)0.10, \(-\)0.15] & [0.40, 0.45] \\ Obstacles2 & [0.55, 0.55] & [0.40, \(-\)0.55] \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Initial position for the agents, targets, and obstacles. Two indicators are usually used to evaluate different deep reinforcement learning models: the convergence speed and the reward value after convergence. To show the effectiveness of the emotional intrinsic motivation module in MADDPG-E, we test its performance with the MADDPG algorithm and MADDPG-ICM algorithm (MADDPG algorithm with curiosity intrinsic motivation). The average reward is as in Figure 5. It can be found that, firstly, compared with MADDPG (blue line), both MADDPG-E (green line) and MADDPG-ICM (red line) finally converge to a significantly higher reward value. The larger the reward value after convergence, the more times the agents search for the target, which demonstrates the effectiveness of the intrinsic motivation. Secondly, although with a similar reward value, we can finally see that MADDPG-E begins to converge at about 6000 episodes, while MADDPG-ICM begins to converge at about 7500 episodes. This proves that MADDPG-E converges faster than MADDPG-ICM, which shows the effectiveness of the intrinsic motivation module in MADDPG-E.  
Figure 4: Multi-agent target search experiment with fixed initial positions. MADDPG-E has a faster learning speed, and the emotional intrinsic module here is useful. Figure 6 shows the mean reward values and standard deviations of the multi-agent system in 6000 episodes of training. We can see that the standard deviation of the MADDPG-E algorithm is the smallest, and the smaller the standard deviation means the better the stability of the algorithm model. This means MADDPG-E in this paper has better learning stability compared with the other two algorithms. In addition, in order to test the performance of the algorithm models during the learning process, a score evaluation index is defined here. In each episode of training, the agents get five points if they find the target and no score if they fail to find the target.