## 3 Basic Design of Multi-Agent Path-Finding
The final loss function can be expressed as Equation (6):  
\[L(\phi,D)=E_{(s,a,r,s^{\prime},d)\sim D}\bigg{[}\Big{(}Q_{\phi}(s,a)-y_{\phi_ {\text{target}}}\Big{)}^{2}\bigg{]} \tag{6}\]  
where  
\[y_{\phi_{\text{target}}}=r+\gamma(1-d)\underset{d^{\prime}}{max}Q_{\phi_{ \text{target}}}(s^{\prime},a^{\prime}) \tag{7}\]In Equation (7), action \(a^{\prime}\) is also the estimated response through the target policy \(\mu_{\phi_{\text{target}}}(s^{\prime})\). Therefore, it can be expressed as Equation (8) below:  
\[y_{\phi_{\text{target}}}=r+\gamma(1-d)\underset{a^{\prime}}{max}Q_{\phi_{\text{target }}}(s^{\prime},\mu_{\phi_{\text{target}}}(s^{\prime})) \tag{8}\]  
Register \(D\) holds a large number of past experiences and consumes a lot of resources since it needs to frequently count all past experiences and update them continuously. Therefore, the action value network is updated by sampling a batch size of experience trace \(B\) from the temporary register. This is shown in Equation (9):  
Figure 3:0  
\(t\) denotes the current trajectory, which is randomly sampled from the register of past experiences. The whole batch experience trajectory \(B\) is obtained from one batch size. The batch size used in this paper is 1024. Neural network parameters are updated with the gradient descent method; its work is represented in Equation (10):  
Figure 3:1  
The gradient value is approached by batch experience trajectory \(B\), where the sum of the mean square deviations is obtained by dividing the batch size. In the deep deterministic policy gradient, the goal of the policy network \(\mu_{\theta}(s)\) is to maximize the action value network, where \(\theta\) is mathematically expressed and is calculated as Equation (11):  
Figure 3:2  
The gradient representation is shown in Equation (12):  
Figure 3:3  
Each action value network \(Q_{\phi}(s,a)\) and policy network \(\mu_{\theta}(s)\) has a target network \(Q_{\phi_{\text{target}}}(s,a)\) and \(\mu_{\theta_{\text{target}}}(s)\), respectively. Once the action value network and strategy network have been updated, the target network needs to be updated at the same time. The target network is updated through Equation (13):  
\[ \begin{split}\phi_{\text{target}}&\leftarrow\rho \phi_{\text{target}}+(1-\rho)\phi\\ \theta_{\text{target}}&\leftarrow\rho\theta_{\text{ target}}+(1-\rho)\theta\end{split}  \tag{13}\]  
In Equation (13), \(\rho=0.01\) is the soft updated rate. A higher \(\rho\) value slows down target network updates, providing smoother policy improvements, while a lower \(\rho\) value speeds up updates but may introduce instability. Even if the target network parameters \(\phi_{\text{target}}\) and \(\theta_{\text{target}}\) of the action value network and the policy network are using the current network parameters \(\phi\) and \(\theta\), the update of their network parameters is gradually completed incrementally. The architecture of the depth deterministic policy gradient is shown in Figure 4. At the beginning of the flowchart, the parameters \(\phi\) and \(\theta\) of the action value network and the policy network are initialized, and the register \(D\) is cleared. The parameters of the target network from the action value network and the policy network are also initialized by \(\phi_{\text{target}}\leftarrow\phi\) and \(\theta_{\text{target}}\leftarrow\theta\) from the current network state. In each training round, the training environment is first reset; the agent obtains the states of the environment. The action \(a\) is selected via the strategy network, and Gaussian noise \(\varepsilon\)[36] is added to improve the ability and effectiveness to optimally explore the space in response to changes in its environment. It should be noted that noise was only used during the training cycle. No noise was taken to improve the exploration after completing the training round. On the other hand, the action value \(a\) was set in such a way that it will not exceed the range in which the agent can normally operate.