## 3 Basic Design of Multi-Agent Path-Finding  
In the design of multi-agent path-finding, it originates from Q-learning, and it is known that if there is an optimal action value function \(Q^{*}(s,a)\), the optimal action a*(s) can be obtained given a state s, as shown in Equation (1):  
\[a^{*}(s)=argmax_{a}Q^{*}(s,a) \tag{1}\]  
Figure 3: Observation information schematic diagram. Figure 2: Architecture diagram of Gazebo simulator for reinforced learning applications. The DDPG algorithm allows the agent to continuously interact with the environment and updates the parameters according to the reward values, which are obtained from the reward function. However, if a continuous action space is used, it is very difficult to obtain the maximum Q value because it is not possible to exhaust all the actions. Approximating the maximum Q through a useful policy is the most efficient and feasible approach, as shown in Equation (2):  
\[max_{a}Q(s,a)\approx Q(s,\mu(s)) \tag{2}\]  
The optimal action value function \(Q^{*}(s,a)\) can be written as Equation (3) using [PERSON]'s equation. \(Q(s,a)\) denotes the optimal expectation value obtained by performing an action in the current state s, where \(Q(s,a)\) is \(s^{\prime}\) the state sampled from the environment, and the distribution of the state transfer probability is denoted as \(P(\cdot|s,a)\). Because future expectations are less relevant to the present moment, a discount rate \(\gamma\) represents the relative importance of future rewards compared to immediate rewards. A higher discount rate prioritizes long-term rewards, while a lower discount rate focuses on immediate gains. \(\gamma=0.95\) is added to reduce the dependence on future expectations. \[Q^{*}(s,a)=\underset{s^{\prime}\sim P(\cdot|s,a)}{\mathbb{E}}\bigg{[}r(s,a)+ \gamma\underset{a^{\prime}}{max}Q^{*}(s^{\prime},a^{\prime})\bigg{]} \tag{3}\]  
The mean squared Bellman Error (MSBE) is used in the design of the action value network to estimate the value of the error of the network, which is with respect to the optimal action value function and is expressed as Equation (4), where \(y_{\phi}\) is the target network of \(Q_{\phi}(s,a)\). r is the reward value, and d is termination status. \[L(\phi,D)=E_{(s,a,r,s^{\prime},d)\sim D}\Big{[}(Q_{\phi}(s,a)-y_{\phi})^{2} \Big{]} \tag{4}\]  
The purpose of the action value network is to calculate the reward value, which is expected to be obtained by performing action A in a state S. It is denoted in Equation (5):  
\[y_{\phi}=r+\gamma(1-d)\underset{a^{\prime}}{max}Q_{\phi}(s^{\prime},a^{\prime}) \tag{5}\]  
in the training process, the target network \(y_{\phi}\) is regarded as the best action value function, and the error value (loss) \(L(\phi,D)\) is referred to as the loss between the current action value network and the target network. The system calculates the mean square deviation between the current action value network \(Q_{\phi}(s,a)\) and the target network for training verification. In this case, the value of d is either 0 or 1 and is used to represent the termination state. The agent will not calculate the reward value after reaching the termination state. Further, D is a register for storing the trajectory of past experiences. The DDPG learns by experience replay, which is similar to the DQN. This is an efficient way to utilize the experience of different rounds of the training cycle through offline learning methodology. A larger replay buffer provides more data for training, potentially improving policy accuracy, but may increase memory usage. The depth deterministic policy gradient also refers to the DDQN, which takes advantage of the target network to avoid the problem of overestimation. The network parameters are those of the target network \(\phi_{\text{target}}\).