## 2 Related Work
### Reinforcement Learning  
The purpose of the reinforcement learning algorithm is to use this procedure to iteratively update the policy \(\pi(a|o)\) so that the agent can receive the maximum reward sum until it approaches the optimal policy \(\pi^{*}\). As the basic method to solve this MDP, reinforcement learning obtains the current observation by observing the environment, selects an action according to the current situation by using the iterative optimization method to maximize the total reward, and updates the strategy in the interaction between the agent and the environment [39]. Reinforcement learning can be divided into three categories. The first type is value-based methods, such as the deep Q learning network (DQN) [40] proposed by [PERSON] et al. and the DRQN proposed by [PERSON] et al. [41], which uses the deep neural network as a function approximation to solve large-scale problems, stores interactive information by maintaining an experience replay buffer, and adds the target network to be especially responsible for constructing the temporal-difference target value, to ensure the stability of training. The value-based method is extremely inconvenient for continuous action processing, and the learning ability is low under certain constraints. Therefore, some researchers have proposed the policy-based method, which does not approximate the value function but the policy. The trust region policy optimization (TRPO) [42] proposed by [PERSON] limits the update step by using Kullback-Leibler (KL) divergence. This method is used to ensure that the differences between the old and new policies are not too large. Proximal policy optimization (PPO) [43] removes the KL penalty item and alternate update of TRPO, and adds the KL divergence regularization part on the basis of the original objective function to represent the difference between the two distributions, which greatly reduces the calculation process of TRPO and improves the training speed. The third method is to combine the above two ideas, that is the actor-critic (AC) algorithm [44]. The algorithm consists of two parts. Actor refers to the policy network, which can obtain the policy distribution through the state input and output the  
Figure 1: Description of the POMDP model. probability value of the corresponding action. This probability distribution is also called the policy. Critic refers to the value function network, which is responsible for evaluating the performance of the policy network and guiding the action of the policy network in the next stage. The AC algorithm combines the advantages of value-based and policy-based methods and can be used in high-dimensional and continuous action space. DPG adopts deterministic policy gradient [45], does not output the probability distribution of actions, but only adopts the work with maximum probability and uses the off-policy method to explore and update. Then, [PERSON] et al. proposed DDPG [32] algorithm by combining DQN and DPG, using the double-actor neural network and double-critic neural network with experience replay to improve the problem of actor-critic convergence. In asynchronous advantage actor-critic (A3C) [46], multi-thread is adopted. The main thread is responsible for updating the parameters of the actor and critic. The secondary thread creates multiple agents to perform asynchronous learning in multiple environments to increase the number of explorations. It summarizes and updates the parameters of the main thread, similar to the experience replay in DQN. In the field of multi-agent reinforcement learning, [PERSON] et al. [47] proposed multi-agent deep deterministic policy gradient (MADDPG) that is suitable for both cooperation and confrontation scenarios. It uses centralized training and decentralized execution to enable agents to learn better cooperation and improve overall performance. [PERSON] et al. [48] proposed counterfactual multi-agent policy gradients (COMA), which also uses centralized training and decentralized execution. The network is a single critic and multiple actors.