## 3 Methods
### The MADDPG-E Algorithm Framework
* \(R(t)\) is the average value of the rewards obtained by \(N\) agents at the time \(t\) and is the overall reward of the multi-agent system in the collaborative target search process. The formula is as follows: \[\left\{ \begin{array}{l}R(t)=\frac{1}{N}\sum\limits_{i}^{N}\left[-0.1\times d _{i}+r_{i}^{em}(t)+r_{i}^{e}(t)\right],\\ d_{i}=\left(\sqrt{(x_{i}(t)-x_{tar}(t))^{2}+(y_{i}(t)-y_{tar}(t))^{2}}\right) \end{array} \right.\] (5) where \((x_{tar}(t),y_{tar}(t))\) is the location of the target at the time \(t\). \(r_{i}^{em}(t)\) is the emotional intrinsic motivation reward of agent \(i\). \(r_{i}^{e}(t)\) is the environmental reward of agent \(i\) at the time \(t\). * \(M\) stands for the memory module, which stores the collected experience with an experience playback array, each of which is a quadruplet \(\{s(t),a(t),s(t+1),R(t)\}\) as follows: \[\left\{ \begin{array}{l}s(t)=\left[\o^{1}(t),\ldots,\o^{N}(t)\right],\\ a(t)=\left[\o^{1}(t),\ldots,\o^{N}(t)\right],\\ s(t+1)=\left[\o^{1}(t+1),\ldots,o^{N}(t+1)\right],\\ R(t)=\left[R_{1}(t),\ldots,R_{N}(t)\right]\end{array} \right.\] (6)where \(o^{N}(t)\) is the observed state of agent \(N\) at time \(t\). \(a^{N}(t)\) is the action chosen by the agent \(N\) at time \(t\). In our algorithm, for \(N\) agents in the system, each agent \(i\) has an \(Actor\) network \(\mu\left(S_{i^{\prime}}\theta_{i}^{n}\right)\) and a \(Critic\) network \(Q\left(S_{i},A_{i^{\prime}};\theta_{i}^{Q}\right)\). The \(Actor\) network is deterministic, and for the deterministic inputs \((S_{i^{\prime}},O_{i})\), the output action \(a^{i}\) is deterministic. The input of the \(Critic\) network is the global state and the actions of all the agents, and the output is a real number. It indicates the degree to which action \(a\) is performed based on the state \(s\). The \(Critic\) networks are used to evaluate all the actions and guide the \(Actor\) networks to make improvements. The role of the \(Actor\) network is to increase the average value of the \(Critic\) network by improving the parameters \(\theta_{i}^{n}\) through training. The gradient of the expected return for agent \(i\) is as follows:  
\[\
abla_{\theta^{i}}J\left(\theta^{i}\right)=\mathbb{E}_{o^{i}\sim N}\left[ \
abla_{a^{i}}Q^{i}\left(o,a^{1},\ldots,a^{N}\right)\
abla_{\theta^{i}}\theta _{i}^{n}\left(o^{i}\right)\left|a^{i}\right.\right] \tag{7}\]  
where \(Q^{i}\) corresponds to the centralized critic of agent \(i\). Its input consists of the agent joint observation \(o=\left(o^{1},o^{2},\ldots,o^{N}\right)\) as well as the chosen specific actions \(\left(a^{1},a^{2},\ldots,a^{N}\right)\) of all the agents. The role of the \(Critic\) network is to conduct centralized training in joint observation and action.The policy input solely consists of the individual observation \(o^{i}\) to choose action \(a^{i}\). Centralized critics for deterministic action policies are optimized with respect to the following loss function. Update the \(i\)-th value network with a TD error so that the value network better fits the value function \(Q(s,a)\). The update of the \(Critic\) network is as follows:  
\[ \begin{cases}\mathcal{L}(\theta_{i})=\mathbb{E}_{o,a,R,\sigma^{ \prime},M}\left[\left(y-Q^{i}\left(o,a^{1},\ldots,a^{N}\right)\right)^{2} \right]\\ y=R+\gamma\mathcal{Q}^{i^{\prime}}\left(o^{\prime},a^{1^{\prime}},\ldots,a^{ N^{\prime}}\right)\left|{}_{a^{i^{\prime}}=\mu^{i^{\prime}}\left(o^{i}\right)} \end{cases}  \tag{8}\]  
where \(\mu^{i^{\prime}}\) corresponds to the deterministic policy of agent \(i\). And \({Q^{i^{\prime}}}\) represents the critic value with delayed parameters for agent \(i\). Both the \(Critic\) target network and the \(Actor\) target network use soft update methods for the parameter:  
\[soft\ update:\left\{ \begin{array}{l}\theta^{n^{\prime}}\leftarrow\tau\theta^ {n}+(1-\tau)\theta^{n^{\prime}}\\ \theta^{Q^{\prime}}\leftarrow\tau\theta^{Q}+(1-\tau)\theta^{Q^{\prime}}\end{array} \right. \tag{9}\]  
Our MADDPG-E algorithm training process takes a centralized training and decentralized execution approach.