## 2 Background  
During multi-agent training, the state of each agent changes, and the environment is unstable from the point of view of other agents. Therefore, traditional reinforcement learning methods, such as Q-Learning or policy gradient, are not suitable for multi-agentenvironments. For this reason, [PERSON] et al. (2017) extended DDPG, a single-agent actor-critic method of deep reinforcement learning, to MADDPG and made it suitable for the multi-agent environment. Its algorithm framework is shown in Figure 1. The most core part of the MADDPG algorithm is that the \(Critic\) part of each agent can obtain the action information of all the other agents for centralized training and decentralized execution. This means, during the training, the \(Critic\) can observe the whole situation and can be introduced to guide the training of the \(Actor\). When testing, the algorithm only uses the \(Actor\) network with local observations to take action.  
In the MADDPG algorithm, the policy set of all the agents is \(\pi_{i}=\{\pi_{1},\ldots,\pi_{N}\}\), and the gradient of the expected return \(J(\theta_{i})=\mathbb{E}[R_{i}]\) for agent \(i\) is  
\[\
abla_{\theta_{i}}J(\theta_{i})=\mathbb{E}_{s\sim p^{\mu},a_{i}\sim\pi_{i}} \big{[}\
abla_{\theta_{i}}\log\pi_{i}(a_{i}|o_{i})Q_{i}^{\pi}(x,a_{1},\ldots,a _{N})\big{]} \tag{1}\]  
where \(p^{\mu}\) is the state distribution. \(Q_{i}^{\pi}(x,a_{1},\ldots,a_{N})\) is a centralized action-value function; it takes the actions \((a_{1},\ldots,a_{N})\) of all the agents and the observed value of all the agents \(x=(o_{1},\ldots,o_{N})\) as the input, and the output is the \(Q\) value of the agent \(i\). For an agent in one state, it may have different actions as choices; however, according to the \(Q\) value calculated by the \(Critic\) network, the agent would like to choose the action with the highest \(Q\) value so as to obtain the maximum reward. The strategy gradient here in the MADDPG system increases the selective probability of actions with a high \(Q\) value and decreases the selective probability of actions with a low \(Q\) value.  
[PERSON] et al. considered \(N\) continuous policies \(\mu_{i}\) and extended them to deterministic strategies. The gradient is written as  
\[\
abla_{\theta_{i}}J(\mu_{i})=\mathbb{E}_{x,a\sim D}\Big{[}\
abla_{\theta_{i }}\mu(a_{i}|o_{i})\
abla_{a_{i}}Q_{i}^{\mu}(x,a_{1},\ldots,a_{N})|_{a_{i}} \Big{]} \tag{2}\]  
where \(D\) is the experience replay buffer, which records the experiences of all the agents. \(o_{i}\) is the observed state of agent \(i\). The centralized action-value function \(Q_{i}^{\mu}\) is updated as  
\[\begin{cases}\mathcal{L}(\theta_{i})=\mathbb{E}_{x,a,\
u,x^{\prime}}\Big{[} \big{(}Q_{i}^{\mu}(x,a_{1},\ldots,a_{N})-y\big{)}^{2}\Big{]}\\ y=r_{i}+\gamma Q_{i}^{\mu^{\prime}}\big{(}x^{\prime},a^{\prime}1,\ldots,a^{ \prime}_{N}\big{)}\bigg{|}_{a^{\prime}=\mu^{\prime}_{j}(o_{j})}\end{cases} \tag{3}\]  
where \(\mu^{\prime}=\big{\{}\mu_{\sigma_{1}},\ldots,\mu_{\sigma_{N}}\big{\}}\) is the set of target policies with delayed parameters \(\theta_{i}^{\prime}\). \(\gamma\) is the discount factor.  
The MADDPG algorithm has the following three characteristics:  
Figure 1: The MADDPG algorithmic framework.  
* The optimal policy obtained by learning only needs to use the local information to take the optimal action.
* The environment and special communication requirements are not needed.
* The algorithm can be used not only in a cooperative environment but also in a competitive environment.