## 4 Path-Finding Design with Multi-Agent Deep Deterministic Policy Gradients
\begin{table}
\begin{tabular}{c c c} \hline \hline
**State** & **Rewards** & **Objective** \\ \hline Achievement of Targets & \(exp(-k1\times\text{dist})\) & Obj 1 \\ \(r_{gal}\) & & \\ \hline Occurrence of collisions & collision\_times & Obj 2 \\ \(r_{collision}\) & & \\ \hline Repulsion Award & \(-max(exp(-k2\times d(P_{\text{agent}},I)))\) & Obj 3 \\ \hline Expert Direction Award & \(\left\{\begin{array}{l}-k3\ \frac{u}{|u|}=\frac{v}{|v|}\\ -k4\ \text{otherwise}\end{array}\right.\) & Obj 4 \\ \hline Stop Rewards & \(-k5\) & Obj 5 \\ \hline \hline \end{tabular}  
Obj 1: The achievement of the target to obtain the reward and set the termination condition for the agent to be reached. The reward function is \(exp(-k1\times\text{dist})\), and the distance between the agent and its target is represented by \(\text{dist}\). \(k1=10\) is used to adjust the degree of convergence between \(\text{dist}\) size and rewards. Obj 2: The reward value of collision, times for a collision, where collision_times is the number of collisions. Obj 3: The reward value of \(-max(exp(-k2\times d(P_{\text{agent}},I)))\) is used to calculate the repulsive force; the closer the distance, the greater the repulsive force, in which 1 is the set of vectors from the agent to other objects. \(k2=0.2\) is the experience value, which can be adjusted for different distance sizes and repulsive force variations. Obj 4: V denotes the expert action vector, and u denotes the agent action vector. When the agent performs an action that is different from the one planned by the expert, it is rewarded poorly. The size of \(k3=0.2\) is chosen mainly in the same value as Obj 2. \(k4=0.4\) is worse, because the direction is different from the expert. Obj 5: When the agent stops moving, it gives a negative reward. \(k5=0.8\) is chosen based on the fact that the severity of stop rewards is 4 times the size of the collision.  
\end{table} 
Table 2: Reward function design. Figure 7: The system architecture of the action network. The current paper is designed to address the five states encountered during the training process of path-finding reinforcement learning. The first is the state of reaching the target point, which is a relatively rare state in the environment during training. If the reward function was designed to be too sparse in this state, it may result in the state being of little benefit in the overall learning. Therefore, this paper used an exponential function to design the reward for reaching the target point, where the agent is rewarded for being within a certain distance. As the distance between the agent and the target is shortened, the agent can obtain a higher reward value. The second state is a collision, which is inevitable during training. This paper did not consider collision as a termination condition because collision gives negative rewards. All of these collisions were used in the experience replay. Setting it as a termination condition would instead make the collision condition sparse. In order to avoid continuous collisions between the agents, it was set in a way that as the number of collisions increased, the heavier the penalty it received. The third state is the repulsive state. To avoid the state of collision being too sparse, the distance between the agent and other objects was regarded as the repulsion force. The repulsive force increases as one gets closer, and the repulsive force is summed up in the reward function. This award not only enhanced the agent's ability to avoid collisions but also densified the state of the collision. The fourth state is the expert direction, and the A* algorithm was used as a guide for the path. The reason for this is that if we used the straight-line distance of the target point as a reference, we may encounter the problem of having a wall in between despite having a very close distance. If this happens, the agent will be required to perform a detour. Therefore, this paper treated the expert's proposed action as a preferred option.