## 1 Introduction  
Reinforcement learning (RL) is a process whereby the agent decides what action to take based on what it sees as the state of the environment. Then, it generates the next moment's state of the environment and the corresponding reward from the environment that provided the basis for the previous action. Thus, the state of the environment obtained by the agent is a very important issue of the reinforcement learning process. DRL mainly uses neural networks to replace the traditional Q-table. When applied to complex problems, Q-tables can be too large to represent or exhaust. After the introduction of the Deep Q-Learning Network (DQN) in 2013, an improved version of the algorithm was proposed in 2015 [1]. One of the improvements focuses on the original version's course of the autonomous navigation of robots [2]. A DQN lets robots operate more efficiently and safely in related outdoor environments; however, one problem of the DQN is that the distribution of states varies considerably, making it difficult to converge the network. Therefore, the target Q-network is added to the learning process. During the learning process, the current Q-network is periodically copied to the target Q-network for sequestration. The development of the Double Deep Q-Learning Network (DDQN) [3] in 2016 adjusted the original Q-value estimation for deep Q-networks. Originally, the value was estimated directly using the Q-network once; however, in the DDQN, the Q-value estimation is conducted using two separate Q-networks preventing the overestimation of Q. Currently, DRL can be divided into value-based and policy-based approaches. The value-based DRL approach is best known as the DQN. It evaluates the value of each action through a value function. Meanwhile, the policy-based DRL method is the earliestreinforcement learning algorithm, and it functions by selecting the best action for the current state. Intensive learning algorithms, such as the actor-critic network (ACN), use the advantages of both approaches [4]. The ACN consists of two networks, the actor network and the critic network. The actor network is a policy-based reinforcement learning network that decides actions based on the current state. Meanwhile, the critic network evaluates the value of the actions taken by the actor network. [PERSON] et al. [5] used soft actor-critic to complete the high dimensionality and continuous action in robotic problems. Automated guided vehicles (AGVs) are directed by the soft actor-critic (SAC) RL algorithm, which uses a type of a sum-tree prioritized experience replay strategy for autonomous navigation applications [6]. A multi-agent DRL machine involves multiple agents that interact with the environment at the same time. However, the actions of each agent affect the state observed by other agents. Many researchers have used multi-agent DRL to accomplish tasks by a sequence-to-sequence multi-agent deep deterministic policy gradient (SMADDPG) algorithm [7]. One study found that the sum-of-costs optimal solution for the classical multi-agent path-finding (MAPF) problem is NP-hard [8]. Many optimal classical MAPF solvers aim to find collision-free paths with minimal cost by searching the joint configuration space or collision resolution space. For example, [PERSON] et al. [9] published PRIMAL, a multi-agent path-finding method based on reinforcement learning and imitation learning (IL). At its core, it uses Asynchronous Advantage Actor-Critic (A3C) [10] as a learning framework for reinforcement learning. Recently, many studies on multi-agent path-finding based on DRL have referred to the success of PRIMAL, such as the GLAS proposed by [PERSON] et al. [11]. The paper used information from area observations as input to the neural network. However, GLAS goes one step further by incorporating a dynamics model into the system architecture. It enables more sophisticated multi-agent path planning for small quadcopters.