## 4 Path-Finding Design with Multi-Agent Deep Deterministic Policy Gradients  
A multi-agent deep deterministic policy gradient was applied in this study. The architecture diagram is shown in Figure 5. We combined the centralized training and decentralized execution architecture for the multi-agent reinforcement learning target for path-finding. In this study, there were N agents under a multi-agent reinforcement learning cycle, where I denotes the agent number. Each agent had its own local observation information. The observation information obtained by the agent is denoted as \(o_{i}\), and the observation information of all agents is denoted as \(\overset{\rightarrow}{o}=o_{1},\ldots,o_{N}\). The action performed by each agent is represented as \(a_{i}\), and the actions performed by all agents are denoted as \(\overset{\rightarrow}{a}=a_{1},\ldots,a_{N}\). The policies of all agents are represented as \(\overset{\rightarrow}{\mu}=\mu_{1},\ldots,\mu_{N}\) with their parameter set at \(\overset{\rightarrow}{\theta}=\theta_{1},\ldots,\theta_{N}\). The transfer function between the observation information and the action set was expressed as \(\mu_{\theta_{i}}:o_{i}\mapsto a_{i}\). N agents interact together in a training environment, and all agents interacting in the environment are viewed as an overall set of states S.  
Although, Markov games capture the intricate interactions between multiple agents for the more realistic modeling of MARL scenarios. Because each agent can only observe its own local observation information, it can also be regarded as a partially observable Markov  
Figure 4: The architecture of the deep deterministic policy gradient. Decision process. But this computational complexity is large. This study treated the multi-agent reinforcement learning as a multi-agent version of Markov Decision processes [37]. The learning difficulty of multi-agent reinforcement learning lies in the fact that the overall system state transfer function can be expressed as Equation (14). \[T:S\times a_{1}\times\ldots\times a_{N}\mapsto S^{\prime} \tag{14}\]  
In the centralized training part, the central critic of the action value network mainly collected the traces of all the agents and evaluated the actions performed by each actor. Actors can obtain gradients from the central critic and update their network parameters. The gradient representation of the actor update is shown in Equation (15). \[\
abla_{\theta_{i}}I(\theta_{i})=E_{\overset{\rightarrow}{o},a\sim D}^{-} \left[\
abla_{a_{i}}Q_{i}^{\overset{\rightarrow}{o}}(\overset{\rightarrow}{o },a_{1},\ldots,a_{N})\
abla_{\theta_{i}}I_{\theta_{i}}(o_{i})\right|_{a_{i}= \mu_{\theta_{i}}(o_{i})}\right] \tag{15}\]  
The central critic's loss function is calculated using Equation (16), where \(y\) is the target function as shown in Equation (17). In Equation (17), \(r_{i}\) denotes the reward value obtained from the environment by agent i when it is in state \(\overset{\rightarrow}{o}^{\prime}\), and all the agents take individual action \(a_{i}^{\prime}\). \[L(\theta_{i})=E_{\overset{\rightarrow}{o},a_{1},\ldots,a_{N},r_{1},\ldots,r_{ N},\overset{\rightarrow}{o}}\left[\left(Q_{i}^{\overset{\rightarrow}{o}}( \overset{\rightarrow}{o},a_{1},\ldots,a_{N})-y\right)^{2}\right] \tag{16}\]  
\[y=r_{i}+\gamma Q_{i}^{\overset{\rightarrow}{o}}(\overset{\rightarrow}{o}^{ \prime},a_{1}^{\prime},\ldots,a_{N}^{\prime})|_{a_{i}^{\prime}=\mu_{\theta_{ j}}^{\prime}} \tag{17}\]  
This paper used multi-agent deep deterministic policy gradients as the trainer for the path-finding reinforcement learning results. In this case, each agent had a strategy network and action value network of the central critic, and each policy had a target policy network. The action value network also had a target network. The central critic's network input was used for all agent observation information and actions. The central critic network is for the action value network. Its purpose is to evaluate the state of the agent and the value of the actions performed by the agent.