## 2 Background  
We use the Markov decision process (MDP) [27] to represent the process of serialized interaction between the agent and the environment in reinforcement learning. In a multi-agent system, MDP can be represented by a five-tuple \(M=(S,A,R,T,\gamma)\), where \(S\) is the state space, \(A\) is the action space, \(R:S\times A\times S\rightarrow\mathbb{R}\) is reward function, \(T:S\times A\times S\rightarrow[0,1]\) is state transition function, and \(\gamma\in[0,1]\) is the discount factor. Agent choose action according to its policy \(\pi_{\theta}:S\to A\), and receives a reward \(r\). Afterward, the environment will transfer to the next state by \(T\). In a partially observable multi-agent environment [28], \(O=O_{1}\times...\times O_{N}\) represent the joint local observation for \(N\) agents. \(A=A_{1}\times...\times A_{N}\) represent the joint action. Meanwhile, the reward function and transition function change to \(R=R_{1}\times...\times R_{N}\), \(T=O\times A\times S\rightarrow[0,1]\). The discount reward is \(R_{t}=\sum_{l=0}^{\infty}\gamma^{t}{}_{t+l}\). Our goal is to find a policy that maximizes the expected collective reward \(R_{\theta}=E_{\tau\sim p_{\theta}(\tau)}[R(\tau)]\). Classical reinforcement learning methods are divided into two categories, one is based on Q-learning [29], and the other is based on policy gradient [30]. Q-learning calculates the value of state-action pair called Q-value:  
\[Q_{\pi}(s,a)=E_{\tau,s^{\prime}}(R|s_{t}=s,a_{t}=a) \tag{1}\]  
Based on the Bellman equation \(Q_{\pi}(s,a)=E_{\pi}[r+\gamma Q(s^{\prime},a^{\prime})]\) the Q-value can update by  
\[Q(s,a)\gets Q(s,a)+a[(r+\gamma max_{a^{\prime}}Q(s^{\prime},a^{\prime}))- Q(s,a)] \tag{2}\]  
[PERSON] proposed the DQN [31] based on Q-learning, which uses a neural network to fetch Q-function. The loss function is  
\[\mathcal{L}(\theta_{i})=E_{s,a,\mathcal{P},\mathcal{I}}[(\tau+\gamma max_{a^{ \prime}}Q(s^{\prime},a^{\prime};\theta_{i}^{-})-Q(s,a;\theta_{i}))^{2}] \tag{3}\]  
The policy gradient algorithm use a neural network to fit policy directly. The gradient is computed by  
\[\
abla_{\phi}\mathcal{I}(\pi_{\phi})=E[\
abla_{\phi}\log\pi(a|s)Q^{\pi}(s,a)] \tag{4}\]The proximal policy optimization (PPO) [32] algorithm uses clip function to limit the magnitude of the gradient update while ensuring the effective update of the policy, and uses experience replay buffer to increase data utilization efficiency. The gradient is computed by:  
\[J^{\theta^{k}}(\theta)=\sum_{s_{t},a_{t}}\min{(\frac{p_{\theta}(a_{t}|s_{t})}{p _{\theta^{k}}(a_{t}|s_{t})}A^{\theta^{k}}(s_{t},a_{t}),clip(\frac{p_{\theta}(a _{t}|s_{t})}{p_{\theta^{k}}(a_{t}|s_{t})},1-\epsilon,1+\epsilon)A^{\theta^{k}} (s_{t},a_{t}))} \tag{5}\]  
\[clip(a,a_{-}min,a_{-}max)= \begin{cases}a_{-}min&\text{if }a<=a_{-}min\\ a_{-}max&\text{if }a>=a_{-}max\\ a&\text{otherwise}\end{cases}  \tag{6}\]  
The above algorithms are classic single-agent reinforcement learning algorithms. In a multi-agent system, a simpler algorithm is to assign each agent an A3C [33] or Q-learning algorithm, like IAC and IQL. However in this way, each agent regards other agents as part of the environment, and does not consider the behavior of other agents when updating, which leads to instability. Therefore, [PERSON]. [34] proposed the MADDPG algorithm and established a centralized critic network to calculate the joint Q value and then update the policy. The gradient formula is:  
\[\
abla_{\theta_{i}}J(\mu_{i})=\mathbb{E}_{x,a\sim\mathcal{D}}[\
abla_{\theta_ {i}}\mu_{i}(a_{i}|o_{i})\
abla_{a_{i}}Q_{i}^{\mu}(x,a_{1},...,a_{N})|_{a_{i}= \mu_{i}(o_{i})}] \tag{7}\]  
The centralized action-value function \(Q_{i}^{\mu}\) is update as:  
\[\mathcal{L}(\theta_{i})=\mathbb{E}_{x,a,r,x^{\prime}}[(Q_{i}^{\mu}(x,a_{1},...,a_{N})-y)^{2}],y=r_{i}+\gamma Q_{i}^{\mu}(x^{\prime},a^{\prime}{}_{1},...,a^{ \prime}{}_{N})|_{a^{\prime}{}_{j}=r^{\prime}_{j}(o_{j})} \tag{8}\]  
Intrinsic reward improves the performance of the RL algorithm and solves the problem of sparse reward [35].