## 4 Solution Based on Multi-Agent Reinforcement Learning  
### Multi-Agent Reinforcement Learning Algorithm  
The MADDPG is used to solve the above Markov game problem in this paper. In order to reach equilibrium in the non-stationary game environment as soon as possible, the MADDPG adopts a centralized training mode, in which additional information can be introduced to improve the stability of the environment. As shown in Equation (28), additional information of other agents is introduced into the training process. Even if the strategies of other agents change (\(\pi_{i}\
eq\pi_{i}\)), the training environment is still stable.  
\[\mathcal{P}\big{(}s^{\prime}|s,a_{1},\dots,a_{N},\pi_{1},\dots,\pi_{N}\big{)}= \mathcal{P}\big{(}s^{\prime}|s,a_{1},\dots,a_{N}\big{)}=\mathcal{P}\big{(}s^{ \prime}|s,a_{1},\dots,a_{N},\pi_{1}\,^{\prime},\dots,\pi_{N}\,^{\prime}\big{)} \tag{28}\]  
Agent \(i\) is composed of the online actor network \(\mu_{i}\) and the online critic network \(Q_{i}\), and their parameters are \(\theta_{i}\) and \(\eta_{i}\), respectively. The experience generated in the interaction with the environment is continuous and does not meet the requirements of an independent and identical distribution of sampling, so replay buffer \(D\) is used to store historical experience, with the capacity of \(N_{e}\). During training, random sampling is conducted in the replay buffer to cut off the correlation between experiences. The objective of reinforcement learning is to find a strategy to maximize reward expectation \(J\).  
\[J(\mu_{i})=E\Big{[}Q_{i}^{\mu}(o,a_{1},\dots,a_{N})\Big{]} \tag{29}\]  
Actor network \(\mu_{i}\) updates its parameters based on the gradient ascent method. The gradient of the objective function is shown in Equation (30):  
\[\
abla_{\theta_{i}}J(\mu_{i})=E_{o,a\sim\mathcal{D}}\Big{[}\
abla_{\theta_{i} }\mu_{i}(o_{i})\
abla_{a_{i}}Q_{i}^{m}(o,a_{1},\dots,a_{N})\Big{|}_{a_{i}=\mu_ {i}(o_{i})}\,\Big{]} \tag{30}\]  
Where the policy gradient is related to the gradient of the critic network, and the online actor network \(\mu_{i}\) achieves improvement with the help of the critic network gradient.  
In order to improve the stability of the learning process, the target actor network (with parameter \(\theta_{i}\,^{\prime}\)) and the target critic network (with parameter \(\eta_{i}\,^{\prime}\)) are introduced. The target network has the same structure as the online network, but the parameter update frequency is different. The online critic network is updated by minimizing the error \(L\) with the target value \(y_{i}\):  
\[L(\theta_{i})=\mathbb{E}_{o,a,r,o^{\prime}\sim\mathcal{D}}\bigg{[}\Big{(}Q_{i} ^{\mu}(o,a_{1},\dots,a_{L})-y_{i}\Big{)}^{2}\bigg{]} \tag{31}\]  
\[y_{i}=r_{i}+\gamma Q_{i}^{\mu^{\prime}}\big{(}o^{\prime},a^{\prime},\dots,a^{ \prime}_{L}\big{)}\Big{|}_{a^{\prime}=\mu^{\prime}_{i}(o^{\prime}_{i})} \tag{32}\]  
where \(o^{\prime}\) is the state observed after the agent executes action \(a\), and \(\gamma\) is the discount factor.  
The soft update strategy is used to update the target actor network and target critic network, as shown in (33) and (34):  
\[\theta^{\prime} \leftarrow\rho\theta+(1-\rho)\theta^{\prime} \tag{33}\] \[\eta^{\prime} \leftarrow\rho\eta+(1-\rho)\eta^{\prime} \tag{34}\]  
where \(\rho<<1\) is the update parameter of the target network, and the soft update is relatively slow.  
Figure 2 shows the interaction process between multiple agents and the environment. \(\mu_{i}\) makes decisions based on the local observations. The action of all agents jointly affects the environment, and each agent also receives corresponding real-time rewards. This transition is stored in the replay buffer of each agent. During training, agents take batch samples from replay buffers. \(Q_{i}\) needs to obtain overall state and action information. \(\mu_{i}\) only needs local observation information.