## 3 Methods  
### The MADDPG-E Algorithm Framework  
The basic idea of MADDPG is its framework of centralized training with decentralized execution. That is, during the training process, the _Critic_ network of each agent collects the state and action information of all the agents, but during the training phase, decisions are made only by each agent's _Actor_ network based on local information for the agent's own actions and states. Due to problems such as a sparse reward and unstable environment, the agent is not motivated to explore, resulting in a low reward and insufficient model convergence. Therefore, the MADDPG algorithm with emotional intrinsic motivation named MADDPG-E is proposed in this paper, and its algorithm framework is shown in Figure 2. In MADDPG-E, a new emotional intrinsic motivation module is designed. The emotional intrinsic motivation reward at each time can be generated according to the environmental stimuli and cognitive states, and such an intrinsic reward as well as the environmental rewards of the agent can then be used as the overall reward of the agent's search process. In this way, not only can the problem of reward sparsity be effectively solved but also collisions can be better avoided. The algorithm proposed in this paper consists of six elements: \(\langle S,O,A,r^{e}(t),R,M\rangle\). The specific meaning of each element is as follow:  
* \(S=\{s_{k}|k=1,2,\ldots,m\,\}\) is the set of the state space, \(s_{k}\) represents the \(k\)-th state of the agent, and \(m\) represents the state number of the agent. All agents share the same state space. Figure 2: The MADDPG-E algorithmic framework. * \(O=\{O_{1},O_{2},\ldots,O_{N}\}\) is the set of the multi-agent observation space, and \(N\) is the agent's number. In each episode, every agent observes the state of the environment through perception, and the agent can obtain the position of obstacles, its own position, the position of other agents, and the position of the target within its detectable range. * \(A=\{A_{1},A_{2},\ldots,A_{N}\}\) is the set of the multi-agent action space, \(A_{N}\) is the action of the agent \(N\), and it is mainly related to speed and direction. Then, the action of the agent \(N\) at time \(t+1\) is expressed as \[\left\{ \begin{array}{l}a_{t+1}^{N}=\left(\alpha(t+1),v(t+1)\right)\\ \alpha(t+1)=\alpha(t)+\beta\\ v(t+1)=v(t)+v^{\prime}\end{array} \right.\] (4) where \(\alpha(t+1)\) is the movement angle of the agent at time \(t+1\), \(\beta\) is the change rate of the agent's motion angle, \(v(t+1)\) is the movement speed of the agent at time \(t+1\), and \(v^{\prime}\) is the acceleration. * \(r^{e}(t)=\left\{r^{1}(t),r^{2}(t),r^{3}(t),r^{4}(t)\right\}\) is the set of environmental rewards. All agents share the same environment rewards set. The agent is rewarded for moving to the target location and punished for colliding with the obstacles or bounds. A dynamic penalty function is set for the collision between agents, which can prevent the occurrence of unsafe states to the greatest extent. By setting environmental rewards, agents can learn to move toward the direction with the largest reward value and adopt the search strategy with the largest cumulative reward to help agents search for the target faster. At each time step, the agent changes its state and receives a reward from the environment. The environmental reward function \(r^{e}(t)\) in this paper is designed as follows: \(r^{1}(t)=10\), if the agent has searched the target. \(r^{2}(t)=-2\), if the agent collides with an obstacle. \(r^{3}(t)=-\lambda\sum\limits_{i=1}^{N}\sum\limits_{j=1,j\
eq i}^{N}\left[ \left\|(x_{i}(t),y_{i}(t))-\left(x_{j}(t),y_{j}(t)\right)\right\|\right]^{-1}\), if an agent collides with another agent. \(\lambda\) is the collision penalty factor, and \((x_{i}(t),y_{i}(t))\) is the positions of the agent \(i\) at time \(t\). \(r^{4}(t)=-10\times(\max(|x_{i}(t)|,|y_{i}(t)|)-0.9)\), if \(max(|x_{i}(t)|,|y_{i}(t)|)\geq 0.9\), which represents the agent colliding with the bounds.