## 3 Simulation
### Simulation Environment
#### 3.1.2 Reward Function Definition  
This article aims at the characteristics of reinforcement learning, combined with heuristic methods to define a distance-based reward function, as shown in Equations (16) and (17).  
The reward function mainly consists of two major modules; the first one is based on the distance of the end point of the reward mechanism, and the second one is based on the distance of the reward mechanism with the target obstacle. The reward mechanism based on the distance can make the unmanned vehicle have better rewards when it is closer to the target point, and negative rewards when it is far away from the target point or when it is loitering; the reward mechanism based on the distance to the target obstacle can make the unmanned vehicle have higher negative rewards when it is closer to the obstacle. The reward function that combines the two enables the unmanned vehicle to reach the target point quickly while staying away from the obstacle.  
\[r_{t}=\left\{\begin{array}{ll}\frac{\left(r+r_{ab}\right)*\left(d_{g}-d_{t }\right)}{d_{g}}&d_{g}\geq d_{t}\\ -1&d_{g}<d_{t}\end{array}\right., \tag{16}\]  
where \(d_{t}=\sqrt{\left(x_{t}-x_{e}\right)^{2}+\left(y_{t}-y_{e}\right)^{2}}\) and \(d_{g}=\sqrt{\left(x_{0}-x_{e}\right)^{2}+\left(y_{0}-y_{e}\right)^{2}}\) represent the distance between the autonomous vehicle and the end point at time t and the distance between the starting point and the end point, respectively. \(x_{t}\) and \(y_{t}\) represent the coordinate  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Action** & **Linear Velocity V** & **Angular Velocity W** \\ \hline
0 & \(1\) m/s & \(-1\) rad/s \\
1 & \(1\) m/s & \(-0.5\) rad/s \\
2 & \(1\) m/s & \(0\) rad/s \\
3 & \(1\) m/s & \(0.5\) rad/s \\
4 & \(1\) m/s & \(1\) rad/s \\
5 & \(0\) m/s & \(0\) rad/s \\
6 & \(-0.5\) m/s & \(-0.5\) rad/s \\
7 & \(-0.5\) m/s & \(0\) rad/s \\
8 & \(-0.5\) m/s & \(0.5\) rad/s \\ \hline \hline \end{tabular}
\end{table}
Table 2: State and action space definition.  
points of the autonomous vehicle in the global coordinate system. \((x_{0},y_{0})\) represents the starting point coordinates, and \((x_{e^{\prime}},y_{e})\) represents the end-point coordinates.  
\[r=\begin{cases}20&\text{Reaching the target point}\\ -0.015&\text{Rotate or remain stationary in place},\\ 0.01&\text{Others}\end{cases} \tag{17}\]  
where \(r_{t}\) represents the immediate reward at time \(t\), which is a function with the distance function as the denominator and the reward function \(r+r_{ob}\) as the numerator. For situations where the autonomous vehicle encounters obstacles, does not move or spins out of the map, or does not meet expectations, a negative reward is given as punishment. The autonomous vehicle is rewarded when it reaches the target point. The farther the distance, the larger the \(|r_{t}|\), the closer the distance, the smaller the \(|r_{t}|\). In other cases, a small negative value is given to allow the autonomous vehicle to complete the target task in as few steps as possible. To keep the autonomous vehicle as far away from obstacles as possible, the dilation radii \(d_{min1}\), \(d_{min2}\), and \(d_{min3}\) are set around the obstacles. This article sets their parameter values to \(-0.3\), \(-0.5\), and \(-1\), respectively. As the distance \(d_{ob}\) between the autonomous vehicle and the obstacle decreases, the penalty value gradually increases to \(-1\), as shown in Equation (18).  
\[r_{ob}=\begin{cases}\begin{array}{ll}\frac{-d_{min3}}{4d_{ob}}&d_{min2}\leq d _{ob}<d_{min1}\\ \frac{-d_{min3}}{2d_{ob}}&d_{min3}\leq d_{ob}<d_{min2}\\ -1&d_{ob}<d_{min3}\\ 0&d_{ob}>d_{min1}\end{array}\end{cases} \tag{18}\]